Good point, that should theoretically be double the decode speed, though not sure how the prefill would differ since it’s mainly compute-bound.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Good point, that should theoretically be double the decode speed, though not sure how the prefill would differ since it’s mainly compute-bound.