Comment by bobjordan

19 days ago

Linters...custom made pre-commit linters which are aligned with your code base needs. The agents are great at creating these linters and then forevermore it can help feedback and guide them. My key repo now has "audit_logging_linter, auth_response_linter, datetime_linter, fastapi_security_linter, fastapi_transaction_linter, logger_security_linter, org_scope_linter, service_guardrails_linter, sql_injection_linter, test_infrastructure_linter, token_security_checker..." basically every time you find an implementation gap vs your repo standards, make a linter! Of course, need to create some standards first. But if you know you need protected routes and things like this, then linters can auto-check the work and feedback to the agents, to keep them on track. Now, I even have scripts that can automatically fix the issues for the agents. This is the way to go.

2 comments

bobjordan

ukuina 19 days ago

Aren't many of those tests? Why define them as linters?

bobjordan 18 days ago

Great question, I let Claude help answer this...see below:

The key differences are:

  1. Static vs Runtime Analysis                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Linters use AST parsing to analyze code structure without executing it. Tests verify actual runtime behavior. Example from our datetime_linter:                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                 
  tree = ast.parse(file_path.read_text())                                                                                                                                                                                                                                                                        
  for node in ast.walk(tree):                                                                                                                                                                                                                                                                                    
      if isinstance(node, ast.Import):                                                                                                                                                                                                                                                                           
          if alias.name == "datetime":                                                                                                                                                                                                                                                                           
              # Violation: should use pendulum                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                 
  This catches import datetime syntactically. A test would need to actually execute code and observe wrong datetime behavior.                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                 
  2. Feedback Loop Speed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  - Linters: Run in pre-commit hooks. Agent writes code → instant feedback → fix → iterate in seconds                                                                                                                                                                                                            
  - Tests: Run in CI. Commit → push → wait minutes/hours → fix in next session                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                 
  For AI agents, this is critical. A linter that blocks commit keeps them on track immediately rather than discovering violations after a test run.                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  3. Structural Violations
  For example, our `fastapi_security_linter` catches things like "route missing TenantRouter decorator". These are structural violations - "you forgot to add X" - not "X doesn't work correctly." Tests verify the behavior of X when it exists.

  4. Coverage Exhaustiveness                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Linters scan all code paths structurally. Tests only cover scenarios you explicitly write. Our org_scope_linter catches every unscoped platform query across the entire codebase in one pass. Testing that would require writing a test for each query.                                                        
                                                                                                                                                                                                                                                                                                                 
  5. The Hybrid Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
  We actually have both. The linter catches "you forgot the security decorator" instantly. The test (test_fastapi_authorization.py) verifies "the security decorator actually blocks unauthorized users at runtime." Different failure modes, complementary protections.                                         
                                                                                                                                                                                                                                                                                                                 
  Think of it like: linters are compile-time checks, tests are runtime checks. TypeScript catches string + number at compile time; you don't write a test for that.