1 billion row challenge in Rust using Apache Arrow

Поділитися
Вставка
  • Опубліковано 4 вер 2024

КОМЕНТАРІ • 19

  • @porlando12
    @porlando12 5 місяців тому +18

    R dev here trying to broaden my horizons and wow I'm blown away with the performance! Keep making these!

  • @azmebengraine1437
    @azmebengraine1437 2 місяці тому +4

    great videos, very informative, just subbed.
    ps: increase the editor font size.

  • @richardbennett4365
    @richardbennett4365 4 місяці тому +4

    Nice video, and this man has an accent similar to Will of SpaceRex.
    I like the presentation. The only thing i caught at the end was again a young man in the field of computer science, the third one (and one is from the UK so its not just North American English), who doesn't use the past participle form with have. He said "I've ran" instead of "I've run."
    It seems to be a phenomenon among younger folks, and I wonder if this grammatical form is just not taught in schools anymore, or is it how people, like his parents or friends are speaking in this region of the world. But, i have noticed Elliott Minns of another UA-cam channel (@Dreams Of Code) also using the past tense form of the verb with an auxiliary and not the past participle form of the verb.
    But, it's the message and content that matters. Good job.

  • @mitchbet
    @mitchbet 3 місяці тому +4

    I'm curious how this works. Is it loading the whole file into memory before doing the aggregation? Or the query planner smart enough that it knows it only needs to load 1 line at a time (and then handles keeps track of the required aggregations accordingly)

    • @LtdJorge
      @LtdJorge Місяць тому +1

      He run it multiple times, so the OS has cached the file in memory already.

  • @blaisepascal3905
    @blaisepascal3905 5 місяців тому +4

    Thank you for this nice video!
    What plugin do you use in VS Code to have the option of running/debugging just above the main function?

    • @josiahparry
      @josiahparry  5 місяців тому +5

      It's just the rust-analyzer! I have it mapped to shift+cmd+enter my keybindings.json file though
      {
      "key": "shift+cmd+enter",
      "command": "rust-analyzer.run"
      }

    • @blaisepascal3905
      @blaisepascal3905 5 місяців тому

      @@josiahparry Thanks!

  • @recs8564
    @recs8564 13 днів тому

    Hey Josiah, what is the code complete thingy in vscode with the stars emoji?

    • @josiahparry
      @josiahparry  11 днів тому

      i'm not sure i know what you're talking about? Do you have a time stamp to refer to it?

  • @joshstat8114
    @joshstat8114 4 місяці тому +1

    I was wondering if there is a linear algebra library in extendr (Rust), like RcppArmadillo

    • @josiahparry
      @josiahparry  4 місяці тому +2

      check out faer! faer-rs.github.io/

  • @richardbennett4365
    @richardbennett4365 4 місяці тому

    Nice brief code, but on my machine I'm getting an error with the pretty: implementation Display statement.

    • @richardbennett4365
      @richardbennett4365 4 місяці тому +1

      I got the code debugged. I just have a Code issue where the buffer for the lines of the Terminal needs increasing, and I think I would also like to `tee` the output to file as well as STDOUT.

    • @josiahparry
      @josiahparry  4 місяці тому +1

      ah so this was an editor issue?

  • @dextrowim
    @dextrowim 2 місяці тому +2

    You do know that the java dudes ended this challenge with less than 2 seconds ? :-) But fair is fair, your code is at least understandable...

    • @Gramini
      @Gramini Місяць тому +1

      And the duration highly depends on the machine. The server for the official challenge has 32 cores IIRC.

  • @DougLopes
    @DougLopes Місяць тому +1

    Impossible to read a single word of this in smartphone

  • @kamertonaudiophileplayer847
    @kamertonaudiophileplayer847 3 місяці тому

    It isn't fair.