Having migrated my application's Python and JS codebases to their typed siblings respectively last year, my 2c is that Python typing feels good and worthwhile when you're in the standard lib, but _awful_ (and net-negative) once you leave "normal Python" for the shores of third-party packages, particularly ones that lean heavily on duck typing (Django and BeautifulSoup both come to mind.)
This is where some of the stuff in the TypeScript ecosystem really shines, IMHO — being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.
My experience has been different: last year I started writing Python again after a long break, and I am yet to regret using types pervasively. If some library has no type definitions, I prefer to have my typed code interact with its untyped code. It is still better than having no types at all. You can sometimes get some useful type safety by annotating your functions with the untyped library's classes.
Since then, I have used established libraries like Beautiful Soup, Jinja, Pillow, platformdirs, psutil, python-dateutil, redis-py, and xmltodict with either official or third-party types. I remember their types being useful to varying degrees and not a problem. I have replaced Requests with the very similar but typed and optionally async HTTPX. My most objectionable experience with types in Python so far has been having to write
Admittedly, I don't use Django. Maybe I won't like typed Django if I do. My choice of type checker is Pyright in non-strict mode. It seems to usually, though not always, catch more and more subtle type errors than mypy. I understand that for Django, mypy with a Django plugin is preferred.
You can also use something like stubgen to generate function definition signatures for dependencies for mypy to validate, then make your own changes to those files with better types if you wish.
I don’t think it’s very scalable, and having the library itself or a stubs package come with types is the only “good”-feeling route, but you at least have a somewhat decent path to still getting it decent without any intervention on the library’s part. It may even be sufficient, if (like in most situations) you only use a few functions from a library (which may in turn call others, but you only care about the ones your code directly touches), and therefore only need to type those ones.
While I don’t disagree, there are a number of additional stub libraries you can install that provide typing for libraries that don’t already have them. I personally find [django-types](https://pypi.org/project/django-types/) to be really well constructed and useful.
I agree. Prior to the introduction of types in Python, I thought I wanted it. Now I hate them. It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. You see extraneous code like x: int = 1 in Python now. Third party libs have bonkers types. This function signature is ridiculous:
> It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me.
Maybe try using an IDE? Without one any language's type system will feel more frustrating than it's worth, since you won't get inline error messages either.
> Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1.
This is called type inference, and as far as I can tell this level of basic type inference is supported by the major python type checkers. If you're seeing people explicitly annotate types on local variables that's a cultural problem with people who are unaccustomed to using types.
As for that function signature, it would be bonkers with or without types. The types themselves look pretty straightforward, the problem is just that they formatted it all on one line and have a ridiculous number of keyword arguments.
> I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me.
This is a reasonable take if you're a solo developer working without an IDE. Though I suspect you'd still find a few missing None checks with type checking.
If you're working on a team, though, the idea is to put type-checking into your build server, alongside your tests, linting, and whatnot.
> You see extraneous code like x: int = 1 in Python now.
This shouldn't be necessary in most cases; Python type checkers are fine with inferring types.
> Third party libs have bonkers types. This function signature is ridiculous:
It is. Part of that is that core infrastructure libraries tend to have wonky signatures just by their nature. A bigger part, though, is that a lot of APIs in popular Python libraries are poorly designed, in that they're extremely permissive (like pandas APIs allowing dataframes, ndarrays, list of dicts, and whatever else) and use kwargs inappropriately. Type declarations just bring that to the surface.
> being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.
Alright, but there's nothing stopping you from having a completely typesafe ORM in python, is there?
Sure, there's isn't really one that everyone uses yet, but the python community tends to be a bit more cautious and slower to adopt big changes like that.
I'm talking about practical limitations, not academic ones. You're not incorrect (and libraries like FastAPI and Pydantic make me confident that the benefits of type-safety will grow throughout the ecosystem) but I am talking about from the perspective of someone considering whether or not to adopt typing within their Python project today.
It did, especially in the late 2013 and early 2014s. But then the type repositories quickly caught up. Python package authors usually shy away from such endeavours, especially those who use kwargs in order to configure large classes. pygann comes to mind.
Using types properly is always annoying. Seeing MyPy report no problems makes it worth it.
I find myself doing a lot of isinstance() and raise TypeError, but that's still a huge win, protecting everything after I've asserted the duck type is what it should be.
I also use beartype for runtime protection.
Typescript is pretty amazing though. I really like how integrated the ecosystem is.
> Maybe we can use LLMs to automatically bring these third party libs up to par?
So, I actually tried this. I tried to use copilot to help generate type stubs for a third party library, hoping to be pleasantly surprised.
Copilot generated reasonable-looking type stubs that were not close enough to correct to be of any value. Even with the full source code in context, it failed to "reason" correctly about any of the hard stuff (unions, generics, overloads, variadics, quasi-structured mappings, weird internal proxy types, state-dependent responses, etc. etc.).
In my experience, bolting types onto a duck-typed API always produces somewhat kludgy results that won't be as nice as a system designed around static typing. So _of course_ an LLM can't solve that problem any more than adding type stubs can.
But really, the answer to "will LLMs fix $hard_problem for us?" is almost always "no", because $hard_problem can rarely be solved by just writing some code.
I was surprised to see the example in the blog. Python actually has come pretty far with types but the blog's example doesn't really highlight it. For structural static typing, something like this is nicer as an example
from typing import Protocol, Tuple, TypedDict
class Foo(TypedDict):
foo: str
bar: int
baz: Tuple[str, int]
baaz: Tuple[float, ...]
class Functionality(Protocol):
def do(self, it: Foo): ...
class MyFunctionality: # not explicitly implemented
def do(self, it: Foo): ...
class DoIt:
def execute(self, it: Foo, func: Functionality): ...
doit = DoIt().execute({ # Type checks
"foo": "foo",
"bar": 7,
"baz": ("str", 2),
"baaz": (1.0, 2.0)}, MyFunctionality())
Protocols and TypedDicts let you do nice structural stuff, similar typescript (though not as feature complete). Types are good enough on python that I would never consider a project without them, and I work with pandas and numpy a lot. You change your workflow a little bit so that you end up quarantining the code that interfaces with third-party libraries that don't have good type support behind your own functions that do. There are other pretty cool type things as well. Python is definitely in much better shape than it was.
Combine all of that with Pyright's ability to do more advanced type inference and its like a whole new language experience.
I would supplement this by suggesting `pydantic` models instead of `TypedDict`s. This library has become a core utility for me as it greatly improves the developer experience with typing/validation/serialization support.
They fulfil different roles. pydantic models would be an alternative to dataclasses, attrs or data-centric classes in general. TypedDict is used when you're stuck with dicts and can't convert them to a specific class.
Yeah they're kind of different. I'm only really talking about TypedDicts because the original post was related to structural typing, which isn't what pydantic does. I do reach for pydantic first myself.
TypedDicts are a really disappointing feature. Typing fails if you pass it a dictionary with extra keys, so you can’t use it for many structural typing use cases.
It’s even more disappointing because this isn’t just an oversight. The authors have deliberately made having additional keys an error. Apparently, this even a divergence from how TypeScript checks dictionaries.
For what it's worth, TypedDict was a bit ahead of it's own time. Python 3.12 is really the turning point for being able to leverage them effectively for stuff like **kwargs[1][2]
from typing import TypedDict, Unpack, NotRequired
class Movie(TypedDict):
name: str
year: NotRequired[int]
def foo(**kwargs: Unpack[Movie]) -> None: ...
TypedDicts are disappointing because you can't partially define a type? That seems like a success
In go you would need to define all fields in your struct and if you needed unstructured data you would have to define a map, which even then is partially typed
Where I really want this is pandas. The community has been smoothing the basic typing story over the last couple of years, which helps with deprecations & basic API misuses. However, I'm excited for shape/dependent typing over dataframe column names, as that would get more into our typical case of data & logic errors.
You might want to check pola.rs then, it's backed by the appache arrow memory models and it's written in rust. All the columns have a defined type and you can easily catch a mistake when loading data
Unless I'm misunderstanding, Arrow solves the data representation on disk/memory, both for pandas and polars, while I'm writing about type inferencing during static analysis, which Arrow doesn't solve.
Having a type checking system respect arrow schemas is indeed our ideal. Will polars during mypy static type checking invocations catch something like `df.this_col_is_missing` as an error? If so, that's what we want, that's great!
FWIW, we donated some of the first versions of what became apache arrow ;-)
I've been hunting down column level typing for a while and did not realise polars had this! That's an absolute game changer, especially if it could cover things like nullability, uniqueness etc.
As far as I can tell, it's runtime, not static, so it won't help during our mypy static checks period?
As intuited by the poster above, we already do generally stick to Apache Arrow column types for data we want to control. Anything we do there is already checked dynamically, such as at file loads and network IO (essentially contracts), and Arrow IO conversions generally already do checks at those points. I guess this is a lightweight way to add stronger dynamically-checked contracts at intermediate function points?
Sort of... the IDE would want the mypy (or otherwise) typings to surface that. Internally, the dataframe library should make it easier for the IDE to see that, vs today's norm of tracking just "Any" / "Index" / "Series" / ... .
There are libraries that support shape checking and I‘ve written a package to combine beartype’s runtime typechecks with jaxtyping‘s shapechecks, see https://github.com/davnn/safecheck
Once accustomed to shape checking it‘s quite a boost in productivity for us, no more fiddling around with invalid dimensions.
I'm a bit new to Python but using it a lot the last few weeks.
People here are suggesting that without an ide typing in Python doesn't make sense. I'm finding that as an emacs user this feels true.
Is anyone using emacs primarily and if so, do you have suggestions on what to do to benefit from Python typing?
Also, I've been struggling with python in the repl. Is there a way to see types in a more dynamic way there? Obviously autocomplete works but I wish a function call would suggest the type. I assume I'm not using things correctly.
I've been using Python since 2.x and I have to admit which each new release I toy with leaving this language behind. Type hints are the latest warts in what should have been a beautiful language. I tend to agree with Charles Lesleifer and others here. Python3 is a mess. https://charlesleifer.com/blog/what-happened/
I tried to do this but got bitten by the fact that Python didn't support varardic type arguments... somehow never thought to just pass the arguments as a tuple!
I'll try and put together a Numpy/JAX wrapper for this, because I've been looking for something that does compile-time shape checking properly for a long time!
Still wish Python's type system was as powerful as Typescript's... it's got potential that it just doesn't live up to.
Having migrated my application's Python and JS codebases to their typed siblings respectively last year, my 2c is that Python typing feels good and worthwhile when you're in the standard lib, but _awful_ (and net-negative) once you leave "normal Python" for the shores of third-party packages, particularly ones that lean heavily on duck typing (Django and BeautifulSoup both come to mind.)
This is where some of the stuff in the TypeScript ecosystem really shines, IMHO — being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.
My experience has been different: last year I started writing Python again after a long break, and I am yet to regret using types pervasively. If some library has no type definitions, I prefer to have my typed code interact with its untyped code. It is still better than having no types at all. You can sometimes get some useful type safety by annotating your functions with the untyped library's classes.
Since then, I have used established libraries like Beautiful Soup, Jinja, Pillow, platformdirs, psutil, python-dateutil, redis-py, and xmltodict with either official or third-party types. I remember their types being useful to varying degrees and not a problem. I have replaced Requests with the very similar but typed and optionally async HTTPX. My most objectionable experience with types in Python so far has been having to write
when I used types-lxml with https://github.com/kovidgoyal/html5-parser. In return I have been able to catch some bugs early and to "fearlessly"refactor code with few or no unit tests, only integration tests. The style I have arrived at is close to https://kobzol.github.io/rust/python/2023/05/20/writing-pyth....
Admittedly, I don't use Django. Maybe I won't like typed Django if I do. My choice of type checker is Pyright in non-strict mode. It seems to usually, though not always, catch more and more subtle type errors than mypy. I understand that for Django, mypy with a Django plugin is preferred.
You can also use something like stubgen to generate function definition signatures for dependencies for mypy to validate, then make your own changes to those files with better types if you wish.
I don’t think it’s very scalable, and having the library itself or a stubs package come with types is the only “good”-feeling route, but you at least have a somewhat decent path to still getting it decent without any intervention on the library’s part. It may even be sufficient, if (like in most situations) you only use a few functions from a library (which may in turn call others, but you only care about the ones your code directly touches), and therefore only need to type those ones.
While I don’t disagree, there are a number of additional stub libraries you can install that provide typing for libraries that don’t already have them. I personally find [django-types](https://pypi.org/project/django-types/) to be really well constructed and useful.
I agree. Prior to the introduction of types in Python, I thought I wanted it. Now I hate them. It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me. Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1. You see extraneous code like x: int = 1 in Python now. Third party libs have bonkers types. This function signature is ridiculous:
https://docs.sqlalchemy.org/en/20/orm/relationship_api.html#...
> It feels like a bunch of rigmarole for no benefit. I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me.
Maybe try using an IDE? Without one any language's type system will feel more frustrating than it's worth, since you won't get inline error messages either.
> Even strongly typed languages like rust have ergonomics to help you avoid explicitly specifying types like let x = 1.
This is called type inference, and as far as I can tell this level of basic type inference is supported by the major python type checkers. If you're seeing people explicitly annotate types on local variables that's a cultural problem with people who are unaccustomed to using types.
As for that function signature, it would be bonkers with or without types. The types themselves look pretty straightforward, the problem is just that they formatted it all on one line and have a ridiculous number of keyword arguments.
10 replies →
> I don’t use an IDE, so code completion or whatever you get for it doesn’t apply to me.
This is a reasonable take if you're a solo developer working without an IDE. Though I suspect you'd still find a few missing None checks with type checking.
If you're working on a team, though, the idea is to put type-checking into your build server, alongside your tests, linting, and whatnot.
> You see extraneous code like x: int = 1 in Python now.
This shouldn't be necessary in most cases; Python type checkers are fine with inferring types.
> Third party libs have bonkers types. This function signature is ridiculous:
It is. Part of that is that core infrastructure libraries tend to have wonky signatures just by their nature. A bigger part, though, is that a lot of APIs in popular Python libraries are poorly designed, in that they're extremely permissive (like pandas APIs allowing dataframes, ndarrays, list of dicts, and whatever else) and use kwargs inappropriately. Type declarations just bring that to the surface.
4 replies →
That signature is ridiculous in any programming language. Types aren't the problem here.
1 reply →
I wouldn't mind all of that if the SQLAlchemy documentation would hide all the types until I mouse over them.
Ditto for vim!
> [...] rust have ergonomics [...]
For starters Rust's official linter, clippy, would also tell you that this function has too many arguments. ;) The default (max) is seven[1].
The above function has 36 named arguments ... That is a code UX wtf with or without type annotations.
[1] https://rust-lang.github.io/rust-clippy/master/index.html#/t...
I've literally never seen anyone put types on trivial variables like that. Maybe your team is just inexperienced with types and/or python?
> being able to have a completely typesafe ORM such as Drizzle (https://orm.drizzle.team/) feels like a Rubicon moment, and touching anything else feels like a significant step backwards.
Alright, but there's nothing stopping you from having a completely typesafe ORM in python, is there?
Sure, there's isn't really one that everyone uses yet, but the python community tends to be a bit more cautious and slower to adopt big changes like that.
I'm talking about practical limitations, not academic ones. You're not incorrect (and libraries like FastAPI and Pydantic make me confident that the benefits of type-safety will grow throughout the ecosystem) but I am talking about from the perspective of someone considering whether or not to adopt typing within their Python project today.
1 reply →
That's fair. Though there's a wave of newer Python packages with great typing support. E.g. packages from https://github.com/encode
If I remember correctly, Typescript felt the same way for quite a long time
It did, especially in the late 2013 and early 2014s. But then the type repositories quickly caught up. Python package authors usually shy away from such endeavours, especially those who use kwargs in order to configure large classes. pygann comes to mind.
4 replies →
Using types properly is always annoying. Seeing MyPy report no problems makes it worth it.
I find myself doing a lot of isinstance() and raise TypeError, but that's still a huge win, protecting everything after I've asserted the duck type is what it should be.
I also use beartype for runtime protection.
Typescript is pretty amazing though. I really like how integrated the ecosystem is.
I think you are missing stub libraries. After installing them its been a breeze
Maybe we can use LLMs to automatically bring these third party libs up to par?
Could be a nice showcase project for Copilot.
> Maybe we can use LLMs to automatically bring these third party libs up to par?
So, I actually tried this. I tried to use copilot to help generate type stubs for a third party library, hoping to be pleasantly surprised.
Copilot generated reasonable-looking type stubs that were not close enough to correct to be of any value. Even with the full source code in context, it failed to "reason" correctly about any of the hard stuff (unions, generics, overloads, variadics, quasi-structured mappings, weird internal proxy types, state-dependent responses, etc. etc.).
In my experience, bolting types onto a duck-typed API always produces somewhat kludgy results that won't be as nice as a system designed around static typing. So _of course_ an LLM can't solve that problem any more than adding type stubs can.
But really, the answer to "will LLMs fix $hard_problem for us?" is almost always "no", because $hard_problem can rarely be solved by just writing some code.
2 replies →
Which language do you refer to when you speak of python's typed sibling?
I take jmduke to mean Python with type annotations.
I was surprised to see the example in the blog. Python actually has come pretty far with types but the blog's example doesn't really highlight it. For structural static typing, something like this is nicer as an example
Protocols and TypedDicts let you do nice structural stuff, similar typescript (though not as feature complete). Types are good enough on python that I would never consider a project without them, and I work with pandas and numpy a lot. You change your workflow a little bit so that you end up quarantining the code that interfaces with third-party libraries that don't have good type support behind your own functions that do. There are other pretty cool type things as well. Python is definitely in much better shape than it was.
Combine all of that with Pyright's ability to do more advanced type inference and its like a whole new language experience.
I would supplement this by suggesting `pydantic` models instead of `TypedDict`s. This library has become a core utility for me as it greatly improves the developer experience with typing/validation/serialization support.
They fulfil different roles. pydantic models would be an alternative to dataclasses, attrs or data-centric classes in general. TypedDict is used when you're stuck with dicts and can't convert them to a specific class.
1 reply →
Yeah they're kind of different. I'm only really talking about TypedDicts because the original post was related to structural typing, which isn't what pydantic does. I do reach for pydantic first myself.
TypedDicts are a really disappointing feature. Typing fails if you pass it a dictionary with extra keys, so you can’t use it for many structural typing use cases.
It’s even more disappointing because this isn’t just an oversight. The authors have deliberately made having additional keys an error. Apparently, this even a divergence from how TypeScript checks dictionaries.
For what it's worth, TypedDict was a bit ahead of it's own time. Python 3.12 is really the turning point for being able to leverage them effectively for stuff like **kwargs[1][2]
[1] https://typing.readthedocs.io/en/latest/spec/callables.html#... [2] https://peps.python.org/pep-0692/
1 reply →
To be fair, that behavior on typescript's part is a major hole for bugs to slip through.
Specifically: absent optional keys + extra keys is fundamentally indistinguishable from miseptl keys.
2 replies →
I'm not sure I understand.
TypedDicts are disappointing because you can't partially define a type? That seems like a success
In go you would need to define all fields in your struct and if you needed unstructured data you would have to define a map, which even then is partially typed
How should extra keys behave in "typed python?"
2 replies →
Where I really want this is pandas. The community has been smoothing the basic typing story over the last couple of years, which helps with deprecations & basic API misuses. However, I'm excited for shape/dependent typing over dataframe column names, as that would get more into our typical case of data & logic errors.
You might want to check pola.rs then, it's backed by the appache arrow memory models and it's written in rust. All the columns have a defined type and you can easily catch a mistake when loading data
Unless I'm misunderstanding, Arrow solves the data representation on disk/memory, both for pandas and polars, while I'm writing about type inferencing during static analysis, which Arrow doesn't solve.
Having a type checking system respect arrow schemas is indeed our ideal. Will polars during mypy static type checking invocations catch something like `df.this_col_is_missing` as an error? If so, that's what we want, that's great!
FWIW, we donated some of the first versions of what became apache arrow ;-)
I've been hunting down column level typing for a while and did not realise polars had this! That's an absolute game changer, especially if it could cover things like nullability, uniqueness etc.
1 reply →
do you have a reference for how to use static typing for polars columns? I haven't seen this in their docs...
Pandera helps with some of this. Check it out -- https://pandera.readthedocs.io/en/stable/
We've used it to great effect.
This is neat, I like the direction!
As far as I can tell, it's runtime, not static, so it won't help during our mypy static checks period?
As intuited by the poster above, we already do generally stick to Apache Arrow column types for data we want to control. Anything we do there is already checked dynamically, such as at file loads and network IO (essentially contracts), and Arrow IO conversions generally already do checks at those points. I guess this is a lightweight way to add stronger dynamically-checked contracts at intermediate function points?
Column misnaming/typo is indeed a problem in pandas. I think a powerful IDE could do the trick though.
Sort of... the IDE would want the mypy (or otherwise) typings to surface that. Internally, the dataframe library should make it easier for the IDE to see that, vs today's norm of tracking just "Any" / "Index" / "Series" / ... .
I wish timezone-naive and timezone-aware Timestamps would be different types.
See this post for a comparison of Python datetime libraries. The datetype, heiclockter and whenever libraries have different types for them.
https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...
There are libraries that support shape checking and I‘ve written a package to combine beartype’s runtime typechecks with jaxtyping‘s shapechecks, see https://github.com/davnn/safecheck
Once accustomed to shape checking it‘s quite a boost in productivity for us, no more fiddling around with invalid dimensions.
I'm a bit new to Python but using it a lot the last few weeks.
People here are suggesting that without an ide typing in Python doesn't make sense. I'm finding that as an emacs user this feels true.
Is anyone using emacs primarily and if so, do you have suggestions on what to do to benefit from Python typing?
Also, I've been struggling with python in the repl. Is there a way to see types in a more dynamic way there? Obviously autocomplete works but I wish a function call would suggest the type. I assume I'm not using things correctly.
You might be interested in `dfdx`, shape checked deep learning in rust: https://github.com/coreylowman/dfdx.
I've been using Python since 2.x and I have to admit which each new release I toy with leaving this language behind. Type hints are the latest warts in what should have been a beautiful language. I tend to agree with Charles Lesleifer and others here. Python3 is a mess. https://charlesleifer.com/blog/what-happened/
That blog post is all very sensible until the drive-by on f-strings at the end, I can't begin to understand the rationale for hating them
I tried to do this but got bitten by the fact that Python didn't support varardic type arguments... somehow never thought to just pass the arguments as a tuple!
I'll try and put together a Numpy/JAX wrapper for this, because I've been looking for something that does compile-time shape checking properly for a long time!
Still wish Python's type system was as powerful as Typescript's... it's got potential that it just doesn't live up to.