980
muskrat's data eng expert's hard drive overheats while processing 60k rows (lemmy.dbzer0.com)
submitted 2 years ago by to c/programmer_humor@programming.dev
292 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 145 points 2 years ago (25 children)

60k isn't that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?

  • permalink
  • fedilink
  • source
  • hideshow 25 child comments
    •
  • [+] 145 points 2 years ago* (last edited 1 year ago) (9 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 60 points 2 years ago (2 children)

    i mean its even excel sized depending on how many columns. This is seriously sad and alarming

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 50 points 2 years ago (4 children)

    Sqlite can easily handle millions of rows. Don't sell it short

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 76 points 2 years ago (3 children)

    A TI-86 can query 60k rows without breaking a sweat.

    If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 12 points 2 years ago*

    Don't know what Elmos minions are doing, but I've written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I'm better now (but I'm not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it's been a while) and it needed conversion to XML (or something similar).

    The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).

    But even then I didn't need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.

    But, as I already mentioned, my hard drive didn't overheat on that load.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 9 points 2 years ago (3 children)

    Seriously - I can parse multiple tables of 5+ million row each... in EXCEL... on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.

    Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 3 points 2 years ago* (last edited 2 years ago) (2 children)

    Excel have a limit of 2^20 rows, something more that 1M. Curious what version of excel are you using for that.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 5 points 2 years ago (1 child)

    You're correct - the standard tabs can only hold roughly 1.2 million rows.

    The way to get around that limitation is to use the Data Model within Power Pivot:

    It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):

    You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.

    The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you're going to build anything for professional use.

    The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it's a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it's a nice way to build reports for end-users that's harder for them to fuck up by deleting a formula or hiding a column.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 4 points 2 years ago

    Oh yes, I remember using power query for a few months once I started working with bigger databases, but I saw that moving to Python would be better carrer wise and never came back to excel to do actual work (but at the end everything get exported to excel)

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 years ago

    My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That's neither really fast, nor cool. But stable.

  • permalink
  • fedilink
  • source
  • parent