Comment by chaps

2 months ago

Hi everyone, I'm the plaintiff in this lawsuit. I'm still working on my companion post for tptacek's post! I'll have it ready Soon TM, but feel free to me any questions in the meantime here.

While you're waiting, check out this older post: https://mchap.io/that-time-the-city-of-seattle-accidentally-...

Matt, you do the Lord's work.

Bear in mind that Matt technically lost this, even with the backing of some of the absolute best civil rights lawyers in the country, Loevy and Loevy, fighting on his behalf. This shows you the absurd difficulty in fighting city hall, especially if you're crazy enough to do it without representation.

The one thing working in our favor is what is proposed in TFA: change the law. Once the state Supreme Court has ruled you're hosed unless you can get an amendment. Illinois has a very strong history of amending its FOIA statute, although a proportion of those changes are to further protect information from disclosure, not always on the side of sunshine.

Another change that needs to happen is strong punishment for bodies who lose these fights. In Illinois this is limited to a "$5000 civil penalty" against the body. What is a civil penalty? It's vaguely defined. They used to throw the money to the plaintiff, but in the later cases I fought they simply awarded the money to the county. As one State's Attorney said to me "I don't care if I lose every case, I just write a check out to myself."

(one final note: be careful what you wish for when you litigate, you can end up with an appellate decision like this that solidifying in law the exact thing you were fighting. It's nobody's fault, but it happens. I ended up with one absurd decision that removed prisoners' rights rather than enhanced them.)

  • A losing public body is also generally on the hook for attorney's fees, which can be considerable. But the general problem here is that the public bodies are all spending someone else's money, so the real deterrent you have is how much of their time you can credibly threaten to eat up with legal actions.

    • That's true, as long as you are represented. I knew one lawyer in Illinois who would sit in FOIA court and take all the non-represented persons aside and offer to take their cases and split the attorney fees 50/50. I believe it isn't strictly above-board, but it is a solution to a problem.

      People don't like being put under oath, so you can somewhat temper a public body's future refusals by deposing them or sticking as many of them on the stand. Especially with depositions, if you aren't represented then you can't be giving any attorney discipline for asking completely outrageous questions to force the deponent to admit crimes etc under oath.

      2 replies →

    •     > so the real deterrent you
          > have is how much of their
          > time you can credibly threaten
          > to eat up with legal actions.
      

      Being threatened with billable hours? They must be terrified.

I don't understand the argument that knowing the column names doesn't help an attacker? Especially in a database that doesn't allow wildcards, doesn't it make things much easier if you know you can do '); SELECT col FROM logins, as opposed to having to guess the column name?

And I don't think I disagree with the court on schema vs. file layouts either. It's not the file layout, but it's analogous: it tells you how the "files" (records) are laid out on the "file system" (database tables). For example, denormalization is very analogous to inlining of data in a file record. The notion that filesystems are effectively databases itself is a well known one too. How do you argue they aren't analogous?

  • You can always `SELECT table_name, column_name, data_type FROM information_schema.columns`, which is part of the SQL standard. https://www.postgresql.org/docs/current/infoschema-columns.h...

    Plus, generally if you have SQL injection, you have multiple tries. You're not going to be locked out after one shot. And there's only so many combinations of `SELECT {id,userid,user_id,uid} FROM {user,users,login,logins,customer,customer}` before you find something useful.

    • > You can always `SELECT table_name, column_name, data_type FROM information_schema.columns`, which is part of the SQL standard. https://www.postgresql.org/docs/current/infoschema-columns.h.

      You can "always" do that? Well I just did that. My database said: no such table: information_schema.columns

      And what if my database had disabled this capability entirely?

      Also, is there anything implying SQL here at all? Can't other databases with injection "capability" have schemas?

      > Plus, generally if you have SQL injection, you have multiple tries. You're not going to be locked out after one shot.

      No, you can't say it with such certainty at all. It really depends on what else you're triggering in the process of that SQL injection. You could easily be triggering something (like a password reset, a payment transaction...) where you're severely limited in your attempts.

      > And there's only so many combinations of `SELECT {id,userid,user_id,uid} FROM {user,users,login,logins,customer,customer}` before you find something useful.

      account, accounts, password, passwords, profile, profiles, credential, credentials, auth, auths, authentication, authentications, authentication_info, authentication_infos, authorization, authorizations, passwd, passwds, user_info, user_infos, login_info, login_infos, account_info, account_infos... should I keep going?

      And these are just the logins/passwords; what if the information of interest was something else, like parking tickets?

      8 replies →

    • That's a good point, has anyone hardened a database by locking out users who select columns that don't exist? Or run other dubious queries? This would obviously interrupt production but if someone is running queries on your db it's probably worth it?

      12 replies →

    • A good DBA would restrict the account so that it can't access the information schema. It's easy to imagine an environment with a vigilant DBA and less vigilant web developers.

      3 replies →

    • Ah so what you're saying is that we ought to rename our logins table to "duckwords" because nobody will ever guess that? Also we should probably store passwords in plaintext but name the column "entercod3" because nobody will think of that. Oh and we should use printf with %s to build our queries right?

    • Being able to inject doesnt mean you get the output of a select. The inject can be on non-select statements.

  • > I don't understand the argument that knowing the column names doesn't help an attacker?

    So Kevin Mitnick supposedly did most of his hacking using "social engineering". He'd call up some person, pretend to be in some other department within their organization, and ask them for some specific bit of information he needed to further his attack (or ask them to change some specific thing that would allow him to further his attack).

    Would knowing the structure of Illinois governmental organizations help someone perform social engineering attacks against them? Yes, absolutely.

    Should Illinois therefore keep the internal structures of their organizations -- the department names and the officials who run them -- secret? No, absolutely not.

    First of all, if an attacker doesn't know them, they'll just use other social engineering attacks to figure them out; i.e., hiding the structure doesn't stop social engineering attacks, it just slows them down. Secondly, the value to the public of being able to navigate governmental structures far outweighs the cost of potential attacks.

    This seems to me to be a direct analog: The "organizational structure" is the "database schema", and the "willingness to help a random person on the phone who seems to know what they're talking about" is the "SQL injection vulnerability". If an attacker knows the schema, their job is faster; but if they don't know the schema, they'll just use attacks to figure out the schema; so keeping it private doesn't stop an attack, only slow it down. And the benefit to the public of being able to issue FOIA requests far outweighs the cost of potential attacks.

  • > And I don't think I disagree with the court on schema vs. file layouts either.

    I disagree that the law should prohibit disclosing "file layouts" but it's pretty clear that the law does block that, and I fundamentally agree with you that schemas are directly analogous to file layouts and thus restricted.

    • A SQL schema literally does not indicate the locations of data inside of a file. In fact, the whole reason schemas exist is to decouple the relationships between table rows and the pages and indexes that store that data. We had relational databases before SQL, and there are non-SQL relational (and non-relational) databases today, but you program them, at the query level, with code that is aware of what tables live where.

      A schema is the opposite of a file layout. A schema is to a file layout what a Google search is to an IP address.

      17 replies →

    • >> And I don't think I disagree with the court on schema vs. file layouts either.

      > I disagree that the law should prohibit disclosing "file layouts"

      Note, the court wasn't ruling what the law should say, only what the law says. At least that's my understanding of it. I certainly wasn't opining on what the law should say.

      1 reply →

    • It seems like an unnecessarily ambiguous term.

      Without additional context, I would interpret the term “file layout” to mean the file and directory structure of an application.

      Such an application could potentially store data as plain files, the names of those files may contain personal or sensitive information.

      2 replies →

  • And this part seems self-defeating:

    > Attackers like me use SQL injection attacks to recover SQL schemas. The schema is the product of an attack, not one of its predicates”.

    If it's the product of an attack, but not the end goal, surely it's of value to the attacker?

    It seems clear to me that the statute does, as worded, in principle allow the city not to disclose the database schema - it would compromise the security of the system, or at the very least, it would for some systems, so each request needs to be litigated individually.

    The proposed amendment sounds like a good way to fix this - is it likely that will pass?

    • Lots of things are "of value". That's not the bar the statute sets. To the extent something isn't per se exempted by the statute (as the outcome of the case established schemas are), the burden is on the public body to demonstrate that disclosure Would jeopardize the security of the system.

      12 replies →

    • > If it's the product of an attack, but not the end goal, surely it's of value to the attacker?

      Well sure, but it doesn't help them attack. That's like arguing that since the bank robber wants dollar bills, dollar bills must be a useful tool for breaking into bank vaults.

      2 replies →

  • If you have an injection friendly application then that is the security problem.

    Say someone hacks the db, is the problem easy to guess table names? The column should never have be called "passwords"?

    Perhaps 30 years ago that would sound good.

    Obscurity should hardly ever be a line of defense. If it is the only defense the problem isn't that it wasn't obscure enough.

    Edit:

    I'll do you one better. If you so much as suggest that obscurity is good security you actually openly invite people to fool around with your applications. The odds holes are to be found are much better than elsewhere.

  • I agree with you. Knowing the exact column names can speed up an attack and, in some cases, make it more feasible.

    Why don’t they just request disclosure of what’s actually stored and allow renaming of the columns? It seems odd that knowing the exact column names would be necessary if the goal is simply to understand what data is being stored and its intended purpose.

    • > Knowing the exact column names can speed up an attack and, in some cases, make it more feasible.

      If I'm looking at a database, I like knowing column names, but I like knowing table names more.

  • >It's not the file layout, but it's analogous...How do you argue they aren't analogous?

    laws don't get to be analogous

    foia request: "I'd like the report the committee prepared about the costs for the new bridge"

    response: "denied. the report contains costs laid out in tables with headings, which while not being schemas are analogous, with schemas not being files but being analogous"

  • Yeah, I think it's still useful info for an attacker. But only if the system was actually developed by amateurs who never heard of parameterized queries.

    I find it a bit bizarre that the city uses "our system was developed with no consideration for security" as a valid defense.

Out of curiosity, could you ask for something like "one row of data from every table in the CANVAS database"?

  • This is a technical solution to a people problem. My reading is that the city doesn’t want to give up this information. If that’s the case, a technical solution wouldn’t work, no matter how easy it is. And given that this has already gone to the Illinois Supreme Court (and lost), the only solution is what is discussed at the end: updating the law.

    • I agree this is something of a technical solution, but the court wasn't interpreting whether you could ask for rows from a database, but whether you could ask for the schema directly. I don't think the court had the option of saying "you can't ask for the schema, but asking for a sample row is ok".

      8 replies →

    • > the only solution is what is discussed at the end: updating the law.

      That, and actually penetrating the data system and subsequently "leaking" parts of it. Which is nearly always illegal, but could be considered a form of "Civil Disobedience" especially if done ethically - e.g. removing sensitive data or leaking only aggregates of the data. Either from outside, or by a whistle-blower.

      I'm not saying "hack the government!". But I am arguing that the pressure of "getting hacked" is like the pressure of protests, blockades, occupying facilities etc, all of which civil disobedience, and often simply illegal too. All are tools in the belts of civilians to keep a government in check. Extracting information that a government is not willing to give but that would benefit the governed, should IMO often be considered such a tool as well.

Kudos to you for enduring through this fight! We can only achieve transparency when people choose not to be complacent. Thank you.

What do you think are the next steps?

  • My first step is to actually finish my post :)

    But after that, getting a reasonable law passed to fix this now-broken nonsense.

What are the administrators of CANVAS hiding?

  • Hard to say. One of my personal drivers for this lawsuit is a tip I received that said that Chicago has a list of vendors whose tickets are dropped in the back-end. When I requested that info, the city said they had no such list. I trust my source, so having schema information could help figure out the extent and if they were lying.

    • Earnest question: If you suspect them of lying on the issue, why would you trust them to release the full schema in response to the FOIA request, and not just omit any possibly incriminating columns?

      10 replies →

    • Well that certainly sounds suspicious. But it could also provide more damming evidence of targeting groups, people skimming the till, bribes to make tickets go away, all sort of fun shenanigans.

      And boy they’re fighting suspiciously hard.

      Good luck.

      2 replies →

  • 'ethnicity' header, 'net_income' header... wouldn't doubt chicago could be cave man enough to do this

Have you tried looking for information from the developer about CANVAS? With any luck the developer has support documentation online that describes CANVAS and maybe you'll be able to narrow down your FOIA request.

  • I think the point of the lawsuit is less about CANVAS schema itself and more about the ability of the government to hide this kind of information from FOIA requests.

Damn, this is impressive. I've been fighting with a state agency since December for 17,000 emails. I don't think I've ever tried to request emails and received zero push-back, but a $33 million estimate just, chef's kiss

Very interesting case! Just one question: to what extent do changes in database schemata fall under FOIA in Illinois? That is, if they should change the database schema to conceal whatever it is they're fighting tooth and nail to hide, are they compelled to retain detailed information about that change? Or can they later present you (should the legislation pass) with a cleaned-up, nothing-to-see-here updated version?

I don't want to take away any steam from your sails but giving bad information in regards to case law shouldn't be taken lightly. Your "expert witness" did you a disservice.

Schema is very much a critical field in terms of AuthZ privileges. Just knowing the structure is not far off from knowing the max entropy a password may hold. In regards to InfoSec, table structure is the recon phase which limits effort and minimizes time. Someone with that much time in security knows DBs will be hacked, not if but when. Time is an incredibly important tool which is why we have expirations on so many authN and authZ windows of attack.

I'm glad that you are challenging them but I believe a credible engineer would have made mince meat of your expert and hurt the rest of us who want to see you successful.

It's possible rewriting certain statutes can help us but there is no company worth its salt that would share DB schema.

  • > Just knowing the structure is not far off from knowing the max entropy a password may hold

    Not if the password is hashed, as it should be. Unless the schema somehow indicates that it uses a hash algorithm such as bcrypt that has a maximum password length. And even then, if they pre-hash the password, the password itself could have more entropy than that. And if there is a maximum password length, then you can probably figure that out via other means, like it being listed in the requirements when you set your password. It does tell you the size of the hash of the password, but if the maximum entropy is sufficiently high, as it should be, then it doesn't really matter; it would still be impractical to brute force.

    > there is no company worth its salt that would share DB schema

    So you are saying that every company with a self-hosted or open source product that uses a database isn't worth their salt? If your DB is running on a customer's infrastructure, that customer will by necessity have access to the schema. And likewise if the source code for a product is publicly available it is trivial to determine the schema.

    • I'm not arguing the complexity of hacking a password, I'm familiar. So instead of responding with rainbow tables or how knowing the schema informs you the location of salts for the salted hash (which is the actual proper way), I'll just point to an example.

      Look at how RSA is implemented. Look at the intentional obscurity of S tables and lack of detailed information.

      There is a reason information is withheld. DB schema is just that, information that increases increases the threat.

      And running a DB on someone's infrastructure doesn't necessarily give you access. You need to read up on AuthN and AuthZ.

      If you listed an open source example I'd take the time to poke holes in your strawman argument but you honestly just need to take a step back and think about what you are really arguing.

      Do you really think not having the schema is as inconsequential as having the schema when attacking something? I mean what is the first step most folks do in reverse engineering? I honestly can't believe I'm having to say this.

They can produce a report using english language labels instead of the db column names. Their argument isn't fact it's vexatious obstenance.

  • As mentioned in the post FOIA tends to only include existing records/information, it doesn't extend to producing new work. So producing a new report would be considered too much work. (But fighting a lawsuit to not reveal the schema is fine )

This older post was such a fantastic read, thanks for sharing your story!

> Normally, a flustered public records officer would just reject a giant request for being for “unduly burdensome”… but this sort of estimate is practically unheard of. So much so that other FOIA nerds have told me that this is the second biggest request they've ever seen. The passive aggression is thick. Needless to say, it's not something I'm willing to pay for!

Welcome to Seattle :-)

  • > that's the second biggest FOIA request I've ever seen!

    -Guybrush, from The Secret of Monkey Island