Hi everyone. This is my first post on Lemmy. I'm a senior Computer Engineering student. And I'm working on my senior design project with my friends.

  • BACKGROUND INFO

My friends have basically near 0 hardware knowledge but are willing to work. I have more experience with computer architecture and digital electronics but still I never made something as complex as this before.

The project is a comparison between hardware solutions to image processing. We will implement 3 types of hardware to process live (or pre recorded) video and output it to vga. Then compare their efficieny.

First one is a CPU core (will implement in system verilog which I have some experience with thank god). It will support RV32IM instructions and my friend will write an assembly program to control it.

Second one is an extension to the first CPU core.

Third one will be a specialized hardware for this purpose.

We have 2.5 months to complete half of the project and the other parts will be completed in next semester.

  • ACTUAL QUESTION

So my plan is to create the CPU and measure its performance in these months and leave the other parts for next semester.

I'm planning on creating a 5 stage pipelined in order CPU. I also want to use wishbone since it's easier. But I have a problem.

First is the fact that videos take up a lot of space and we don't have enough RAM (Basys 3). We can buy an Arty S7 50 for it's DDR3 but the controller for it is AXI. I don't have time to learn AXI and I could not find any open source bridges that would work between that and WB B3.

One other option might be to send and process the video from an external camera or my laptop. But how can I make this work? USB is not nearly fast enough for 640x480 24fps.

Note: We won't use any AI in this project. So the time is more crucial.

Any tips? Should we buy a new board or will streaming work? Maybe we can use ethernet port? Idk.

Thanks.

all 9 comments

sorted by: hot top controversial new old
[–] 4 points 4 hours ago* (last edited 4 hours ago)

Outputting to VGA with an FPGA is fairly easy, but depending on what you mean by "image processing", this may be too big of a project given you're the only hardware engineer or maybe all you mean is to take the video input and output it to VGA without any further processing. You did mention wishbone, so I guess if you're planning to reuse a bunch of Open Cores, that could simplify some things.

How much RAM do you have on your dev board? How long of a video do you want to do?

USB2 is fast enough for uncompressed 640x480 @ 24 fps @ 24bit color: 640 * 480 * 24 * 24 = ~177Mbit/s (plus misc overhead) USB isn't so bad to get working on an FPGA dev board (I didn't see you mention which you're using) as it probably has a transceiver chip already along with drivers available. Even if not, you could add a cheap USB interface chip (FTDI makes some popular ones, but there's many others like CP2102, etc). I'd not try to do more modern USB (3+) as it's not needed, but also higher speeds bring more problems. I'd actually try lower quality video (lower res, framerate, and not 24bit color). As you increase the amount of data you need to transfer, you have more potential for issues related to buffering the video (and the extra memory required) or timing requirements (ensuring the video is arriving by the time it needs to, but not too much before).

If I'm understanding your main concern, it's how to get the video data in to your hardware? Generally:

  • camera module (that outputs uncompressed image/pixel data, idk how difficult these are to get running, drivers often abstract a lot of complexity)
  • video stored on memory chip like an eeprom (may be impractical given size)
  • storage device like SD, compact flash, or even an IDE hard drive (idk how difficult SATA is to work with). if you go this route, I recommend avoiding file systems.
  • stream over USB2 from a laptop (whether it's pre recorded file, built in camera, some USB camera or capture device kind of doesn't matter from your hardware perspective)

Other communication interfaces (like ethernet) are probably going to be a lot more effort although I've seen a serial to ethernet interface that worked similarly to those UART <-> USB chips. idk cost or how easy they are to use though.

EDIT: I see dev board is Basys 3 (I didn't realize that's what that was, I thought that was a camera). With only 1.8Mbits of memory, you're going to need to add memory (could be complicated), drastically reduce video quality and/or have some pretty strict timing requirements for the image data entering your hardware and maybe a camera module is the way to go at that point (vs USB from laptop, but I'm thinking flash memory like SD card is going to be more practical). You had mentioned designing a CPU, so I assume you intend to run software on this, which is also going to need to use memory. Depending on what that software wants to do, adding more memory may be a hard requirement. If you're just taking the video and outputting to VGA, you wouldn't even need the CPU and could probably get it to work without extra memory, but without being able to buffer much video data, you still have to ensure your video input meets timing requirements and I'm leaning away from streaming from a laptop due to that I thought you'd be able to buffer more data.

Also this dev board only does 12bit VGA output, so you can reduce your video color bit depth, which will help some.

  • source
  • [–] 3 points 5 hours ago (1 child)

    Don't forget that you can and should take advantage of the faculty at your university/college. That's what they're there for.

    Source: I'm in a similar position and constantly need to remind myself to not be shy about asking for help and guidance.

  • source
  • hideshow 1 child comment
  • [–] [S] 3 points 5 hours ago

    I would but ever since the AI boom every single professor at my unis computer engineering department has gone full slop. They only work on AI projects and use LLMs frequently. I feel like I'm on my own with this lol.

    I might try the EEE department though.

  • source
  • parent
  • [–] 5 points 6 hours ago (1 child)

    Honestly, I’d probably just focus on getting the CPU working first, especially since you only have 2.5 months 😭😭😭😭 Have you thought about using individual frames instead of streaming the whole video? Maybe starting with a lower resolution or grayscale would make things easier, at least for testing the CPU and seeing how it performs. Ethernet could be an option too, but I feel like setting all that up might end up being another headache 🫠. I’d stick with the Basys 3 for now and see how far you can get before buying another board. You can always upgrade later if you really need to. Also, the project sounds pretty cool! Would love to see how the CPU compares to the dedicated hardware once it’s done 😭💯

  • source
  • hideshow 1 child comment
  • [–] [S] 2 points 6 hours ago

    Lol yeah it's a tight deadline. I made CPUs before but they were small projects for my custom ISA (which has like 10 instructions) and they didn't have a bus interconnect.

    This is a huge step up but could be a learning experience. I will definetly post it once it's done (hopefully).

  • source
  • parent
  • [–] 4 points 7 hours ago (3 children)

    That's quite a senior design project. I would look into existing camera interfaces, it looks like mipi is common. https://www.displaywiki.com/knowledge/posts/mipi-interface-basics/

    You probably don't want to implement Ethernet or USB, those are very big interfaces with a lot of overhead.

  • source
  • hideshow 3 child comments
  • [–] [S] 2 points 6 hours ago (2 children)

    Thanks.

    I would need a MIPI camera board I'm guessing?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 4 hours ago

    I think you should look in to MIPI from a hardware protocol perspective before committing to that route. The issue isn't the physical connector, but the complexity of the protocol. I found some open MIPI cores. idk if you're allowed to use open cores, but if they fit in your FPGA (with whatever other logic you need for your CPU and glue logic), then that may work, but if I were a student I'd probably be looking for a simpler interface where I can have the FPGA control the flow of data (like request 1 row of pixels at a time given the memory constraints).

  • source
  • parent
  • [–] 2 points 6 hours ago

    Yes, but it looks like the connector is a tiny thing, I don't know what you have available on your fpga board. Play around with the filters on digikey/mouser/whatever electronics distributor you use to find the part with the least effort to connect, as the interface might be enough of a task as it is.

  • source
  • parent