Introduction If you can write a for-loop, you can do statistics. - Jake Vanderplas A lot of developers shy away from problems which involve statistics or probability. Which is shameful since in today’s data-rich environment, you can gain a lot of insights from data. In this blog post, I’ll show you how to write a simulation tool which requires no knowledge in statistics or probability. Simulations are easy to write and can be a very effective tool in research.
Continue reading

Introduction I prefer to use relational (SQL) databases in general since they provide several features that are very useful when working with data. SQLite is a great choice since the database is a single file, which makes it easier to share data. Even though it’s a single file, SQLite can handle up to 281 terabytes of data. SQLite also comes with a command line client called sqlite3 which is great for quick prototyping.
Continue reading

Introduction Every single company I’ve worked at and talked to has the same problem without a single exception so far - poor data quality, especially tracking data. Either there’s incomplete data, missing tracking data, duplicative tracking data. - DJ Patil I spend a lot of my time digging into data at various companies. Most of the time I’m surprised by what I see and so are the engineers and analysts that work at these companies.
Continue reading

Series Index Python and Go: Part I - gRPC Python and Go: Part II - Extending Python With Go Python and Go: Part III - Packaging Python Code Python and Go: Part IV - Using Python in Memory Introduction In a previous post we used gRPC to call Python code from Go. gRPC is a great framework, but there is a performance cost to it. Every function call needs to marshal the arguments using protobuf, make a network call over HTTP/2, and then un-marshal the result using protobuf.
Continue reading

Series Index Python and Go: Part I - gRPC Python and Go: Part II - Extending Python With Go Python and Go: Part III - Packaging Python Code Python and Go: Part IV - Using Python in Memory Introduction In the previous post we compiled Go code to a shared library and used it from the Python interactive shell. In this post we’re going to finish the development process by writing a Python module that hides the low level details of working with a shared library and then package this code as a Python package.
Continue reading

Series Index Python and Go: Part I - gRPC Python and Go: Part II - Extending Python With Go Python and Go: Part III - Packaging Python Code Python and Go: Part IV - Using Python in Memory Introduction In the previous post we saw how a Go service can call a Python service using gRPC. Using gRPC to connect a Go and Python program together can be a great choice, but there’s a complexity price that goes with it.
Continue reading

Series Index Python and Go: Part I - gRPC Python and Go: Part II - Extending Python With Go Python and Go: Part III - Packaging Python Code Python and Go: Part IV - Using Python in Memory Introduction Like tools, programming languages tend to solve problems they are designed to. You can use a knife to tighten a screw, but it’s better to use a screwdriver. Plus there is less chance of you getting hurt in the process.
Continue reading