you are viewing a single comment's thread
view the rest of the comments
[–] 22 points 1 year ago (3 children)

my favorite awk snippet is !x[$0]++ which is like uniq but doesn't care about order. basically, it's equivalent to print_this_line = line_cache[$current_line] == 0; line_cache[$current_line] += 1; if $print_this_line then print $current_line end.

really useful for those long spammy logs.

  • source
  • hideshow 3 child comments
  • [–] 3 points 1 year ago (1 child)

    Oh that's very interesting. I usually do sort --unique or sort [...] | uniq if I need specific sorting logic (like by size on disk, etc).

  • source
  • parent
  • hideshow 1 child comment
  • [–] 6 points 1 year ago*

    Looking at the above awk snippet, it'll retain order, though. So, sort will normally change the order. The awk snippet won't, just skip occurrences of a given line after the first. Depending upon the use case, that order retention could be pretty desireable.

  • source
  • parent