Institutionen för systemteknik

Transkript

1 Institutionen för systemteknik Department of Electrical Engineering Examensarbete Evaluation of partial reconfiguration for FPGA debugging Examensarbete utfört i Datorteknik, Tekniska högskolan vid Linköpings universitet av Jacob Siverskog LiTH-ISY-EX--10/4393--SE Linköping 2010 Department of Electrical Engineering Linköpings universitet SE Linköping, Sweden Linköpings tekniska högskola Linköpings universitet Linköping

2

3 Evaluation of partial reconfiguration for FPGA debugging Examensarbete utfört i Datorteknik vid Tekniska högskolan i Linköping av Jacob Siverskog LiTH-ISY-EX--10/4393--SE Handledare: Examinator: Andreas Ehliar isy, Linköpings universitet Andreas Ehliar isy, Linköpings universitet Linköping, 18 June, 2010

4

5 Avdelning, Institution Division, Department Division of Computer Engineering Department of Electrical Engineering Linköpings universitet SE Linköping, Sweden Datum Date Språk Language Svenska/Swedish Engelska/English Rapporttyp Report category Licentiatavhandling Examensarbete C-uppsats D-uppsats Övrig rapport ISBN ISRN LiTH-ISY-EX--10/4393--SE Serietitel och serienummer Title of series, numbering ISSN URL för elektronisk version Titel Title Evaluation of partial reconfiguration for FPGA debugging Författare Author Jacob Siverskog Sammanfattning Abstract Reconfigurable computing is an old concept that during the past couple of decades has become increasingly popular. The concept combines the flexibility of software with the performance of hardware. One important contributing factor to the uprising in popularity is the presence of FPGAs (field-programmable gate arrays), which realize the concept by allowing the hardware to be reconfigured dynamically. The current state of reconfigurable computing is discussed further in the thesis. Debugging is a vital part in the development of a hardware design. It can be done in several ways depending on the situation. The most common way is to perform simulations but in some cases the fault-finding has to be done when the design is implemented in hardware. In this thesis a framework concept is designed that utilizes and evaluates some of the reconfigurable computing ideas. The framework provides debugging possibilities for FPGA designs in a novel way, with a modular system where each module provide means to aid finding a specific fault. The framework is added to an existing design, and offers the user a glimpse into the design behavior and the hardware it runs on. One of the debug modules will be released separately under a free license. It allows the developer to see the contents of the memories in a design without requiring special debugging equipment. Nyckelord Keywords fpga, partial reconfiguration, dynamic reconfigurability, reconfigurable computing, debugging

6

7 Abstract Reconfigurable computing is an old concept that during the past couple of decades has become increasingly popular. The concept combines the flexibility of software with the performance of hardware. One important contributing factor to the uprising in popularity is the presence of FPGAs (field-programmable gate arrays), which realize the concept by allowing the hardware to be reconfigured dynamically. The current state of reconfigurable computing is discussed further in the thesis. Debugging is a vital part in the development of a hardware design. It can be done in several ways depending on the situation. The most common way is to perform simulations but in some cases the fault-finding has to be done when the design is implemented in hardware. In this thesis a framework concept is designed that utilizes and evaluates some of the reconfigurable computing ideas. The framework provides debugging possibilities for FPGA designs in a novel way, with a modular system where each module provide means to aid finding a specific fault. The framework is added to an existing design, and offers the user a glimpse into the design behavior and the hardware it runs on. One of the debug modules will be released separately under a free license. It allows the developer to see the contents of the memories in a design without requiring special debugging equipment. v

8

9 Acknowledgments I would like to thank my supervisor and examinator Andreas Ehliar for giving me the opportunity to do this final year project. Without his experience, advices and help it would not have been possible to finish it. I would also like to thank my parents, you are always there when you are needed. vii

10

11 Contents 1 Introduction Background Reconfigurable computing Field-programmable gate array Debugging FPGA designs Purpose Caveats FPGAs Virtex Configurable logic blocks Block RAMs ICAP Other features Configuration of Virtex devices The Xilinx toolchain User constraints file Partial reconfiguration Current state Applications using partial reconfiguration The Xilinx toolchain and partial reconfiguration Technical details Difference based partial reconfiguration Module based partial reconfiguration ICAP for partial reconfiguration Bus macros Troubleshooting Simulation Functional simulation Post synthesis simulation Post place-and-route (PAR) simulation Logic analyzer Internal logic analyzer ix

12 x Contents External logic analyzer Simulating PR designs Conclusion Motivation for a debug framework Use cases Partial reconfiguration Debug framework idea and implementation Introduction System overview Debug modules Implementation details Microcontroller Program memory UART ICAP PRR Design flow XDL Partial reconfiguration Storage of PRM bitstreams Resource usage Block RAM debug tool Description Block RAMs and ICAP Block RAM data in the bitstream Addressing scheme BRAM data segments Bitstream offset Mapping of Block RAMs Implementation Design flow ICAP controller ICAP interface The block RAM name & address extraction script User manual Adding the tool to a design Using the tool Results Partial reconfiguration CRC errors Placement/routing issues Debug framework XDL

13 Contents xi Placement collisions Logic analyzer Feasibility of the framework Conclusion BRAM debug tool An alternative approach Future work Device compatibility Additional modules Software BRAM debug tool Device support Parity bits Block RAM names Modifying the contents Move the user interface to the host computer Support for larger memories Preventing the UUT from accessing the BRAMs Optimizations Logic analyzer Measuring several sets of signals Conclusions Partial reconfiguration Debug framework BRAM debug tool Bibliography 61 A Block RAM bit locations 63 B Virtex-4 BRAM data bit mapping procedure 65 C Example design: configuration data readback using ICAP 67

14

15 Abbreviations ASIC BRAM CLB CPU DCM DSP FAR FPGA HDL ICAP I/O ISE LUT MCU MUX NCD NGD PCF PR PRM PRR RTL Application specific integrated circuit Block RAM Configurable logic block Central processing unit Digital clock manager Digital signal processing Frame address register Field-programmable gate array Hardware description language Internal configuration access port In/out Integrated Software Environment Lookup table Microcontroller unit Multiplexer Native circuit description Native generic database Physical constraints file Partial reconfiguration Partially reconfigurable module Partially reconfigurable region Register transfer level 1

16 2 Contents RAM UCF UART UUT VHDL VHSIC Random access memory User constraints file Universal asynchronous receiver/transmitter Unit under test VHSIC hardware description language Very high speed integrated circuit

17 Chapter 1 Introduction 1.1 Background Reconfigurable computing The concept of reconfigurable computing was first formulated 1960 in a conference paper[3] by Gerald Estrin, where a system with one fixed module and several reconfigurable modules is presented. The idea is that each reconfigurable module is optimized for a certain task, and that they can be changed dynamically while the fixed part of the system is actively running without interruption. It would allow much higher performance for the specific accelerated tasks, but still with a good amount of flexibility. This concept of changing the hardware dynamically is analogous to context switching in software, where a processor provides multitasking by changing the actively running task. It blurs the traditional distinction between hardware and software, i.e., the established view on hardware is that it is fixed, while software is seen as more dynamic. Another advantage is the possibility of saving hardware resources and power consumption, by only having the hardware that is currently used active Field-programmable gate array A field-programmable gate array (FPGA) is a configurable integrated circuit. The FPGA contains programmable logic, memory, interconnect and other parts that are configured by the designer. In most cases, the hardware design is defined with a hardware description language (HDL), such as Verilog or VHDL, which in several passes is translated into a bitstream that is used to configure the FPGA. Once configured, the FPGA behaves as the specified digital system. Most FPGAs today are SRAM based, which gives them the ability to be reprogrammed many times. FPGAs offer shorter development times and better flexibility compared to ASICs (application specific integrated circuits), properties that help reduce the time to market and make them a viable option for many applications. 3

18 4 Introduction (Dynamic) partial reconfiguration Current versions of Xilinx and future versions of Altera devices[9] have the ability to be partially reconfigured, i.e., the possibility to perform glitchless reconfiguration of certain parts of the FPGA while the other parts are actively running. It is even possible to do this reconfiguration with logic in the FPGA itself, and send the configuration data through an internal port (called internal configuration access port, ICAP, in Xilinx devices). This is a realization of the concept described in section The partial reconfiguration technique can enable area reductions which have the potential to bring down the cost of a system by allowing usage of a smaller FPGA than previously needed, since only the hardware that is currently used have to be part of the configuration. The possibility to modify the hardware design dynamically also increases the flexibility of the devices. However, even though the hardware devices have supported this feature for around a decade, it is not very widely used Debugging FPGA designs There is a great need of being able to debug hardware designs, both during development but also when a system is running. Debugging while developing a design is well supported with a wide range of techniques. In some cases the debugging has to be performed when the design is running in the FPGA. In these cases it is not possible to use those techniques. The availability of tools for this purpose is not as extensive as for tools to be used during development. Clearly there is a need for such tools, which enables the developer to get a glimpse into the internal behavior of the hardware. 1.2 Purpose The purpose of this thesis is to investigate the current state of partial reconfiguration, and attempt to find an explanation to why it is not as widely used as one might expect. An application that utilizes partial reconfiguration will be created for demonstration purposes. It should exploit the partial reconfiguration technique and/or hardware, preferably in a novel manner. 1.3 Caveats The thesis is almost exclusively based on Xilinx devices in general, and the Virtex- 4 family in particular. There are two major causes for this: first, Xilinx is the only major FPGA vendor that currently is manufacturing devices which support partial reconfiguration, second, a Xilinx Virtex-4 XC4VLX25 device was the FPGA primarily used for development throughout the thesis. There are, however, merely details that differ between the devices belonging to the Virtex-4 family.

19 Chapter 2 FPGAs This chapter aims to provide a more technical view of the FPGAs needed in order to understand the details of the following chapters. A basic knowledge of FPGAs is assumed. 2.1 Virtex-4 The Xilinx Virtex-4 is an older generation of FPGAs, unveiled in Since then, two newer families (Virtex-5 and Virtex-6) have been released by Xilinx targeting the same market segment. However, the Virtex-4 is still widely used and due to what was available hardware-wise for this thesis, it is the device that is presented here Configurable logic blocks The configurable logic blocks (CLBs) are the main logic resource in Xilinx devices. A single CLB contains four slices. A slice consists among other things of two 4- input LUTs (lookup tables) and two flip-flops. Figure 2.1 illustrates one half of a slice. The LUT can realize any four-input Boolean function, and several LUTs in a CLB can be combined to provide LUTs with up to eight inputs. Some of the LUTs can also be used as a memory resource called distributed RAM. More details about the CLBs can be found in [17]. 4-input LUT clock D flip-flop Figure 2.1: A simplified view of a half portion of a slice in Virtex-4 5

20 6 FPGAs Block RAMs The block RAMs are dedicated storage elements available for the designer. Each block RAM is a dual-port memory that can store up to 18 Kbits of data. Dual-port means that for instance two writes or two reads can be performed simultaneously. The memories can be configured in several ways, where the data width spans from 1 to 36 bits. Even the two data ports of a single memory can be of different widths. Two adjacent BRAMs can also be combined into a larger memory with doubled storage capacity ICAP ICAP is an internal configuration port available in Xilinx Spartan-3A, Virtex-II, and later FPGA families. It is similar in function to SelectMAP, but it is located internally on the FPGA. It is possible to instantiate the ICAP interface in the RTL (register transfer level) code and thus get access to the FPGA configuration 1 data directly from the device. This internal access allows, among other things, reconfiguration, readback of configuration data and reading of internal registers in the device. Features that before only have been available from outside the chip Other features Other features of the Virtex-4 include digital signal processing (DSP) blocks for efficient arithmetic operations (such as multiplications) and digital clock managers (DCMs) for handling several clock domains. Some devices come with integrated PowerPC processors as well. 2.2 Configuration of Virtex devices The FPGA configuration data is located in a bitstream generated by the bitgen tool (part of the Xilinx ISE toolchain). This bitstream describes, among other things, contents of the BRAMs, how the CLBs and the other parts should be configured, and how they should be interconnected. In the normal case, the entire FPGA is programmed at once, but it is also possible to create bitstreams that configures only parts of the device (a feature used while utilizing partial reconfiguration). The smallest addressable segment of the configuration memory space is a frame. Each frame consists of 41 (Virtex-4) 32-bit words in the bitstream, which corresponds to for instance the configuration of 16 CLBs ([7], [18]). It is possible to get the bitstream in a readable form ( rawbits format) by calling bitgen with the -b switch, which eases analysis of the data. A small part of a bitstream in the rawbits format can be seen in the left part of figure 7.2. In the Virtex-4 case, a bitstream configuring the entire device consists of hundreds of thousands of lines of 32-bit words. 1 See appendix A in [8] for a note on the use of the term configuration.

21 2.3 The Xilinx toolchain The Xilinx toolchain The Xilinx ISE (integrated software environment) is a suite of programs for translating a design into a bitstream. It also contains a tool (impact) used to configure the FPGA with the bitstream, as well as other tools for analysis and modification of the design. Figure 2.2 shows a possible design flow using the Xilinx toolchain. A short description of each step of the complete flow from RTL code to a configured FPGA: xst Synthesizer, reads RTL code, compiles it and outputs an NGC netlist ngdbuild map par bitgen impact Works basically like a linker, outputs an NGD netlist from the NGC netlist and the user constraints file Maps the primitives in the netlist to the hardware. Translates the NGD file into an NCD file, and outputs constraints into a PCF file. Places and routes the design in the NCD file with the constraints found in the PCF file. It also performs static timing analysis. Generates a bitstream from the placed and routed NCD file. It also performs a design rule check (DRC) to assert a functional design. Configures the FPGA with the generated bitstream User constraints file The user constraints file, UCF, is a text file that defines constraints on the logical design. In this context, constraints can for instance be timing constraints, placement constraints or pin locations of external signals.

22 8 FPGAs RTL source code Constraints (UCF) xst NGC ngdbuild NGD map NCD (mapped) Constraints (PCF) par NCD (routed) bitgen Bitstream Figure 2.2: Overview of a design flow from RTL code to bitstream using the Xilinx toolchain.

23 Chapter 3 Partial reconfiguration 3.1 Current state Applications using partial reconfiguration There are a wide range of documented applications that use partial reconfiguration, PR, in some way. A typical (and the most common) application has a system consisting of a CPU, some logic for interfacing with the ICAP port (in general the interfaces provided by Xilinx are used), and several partially reconfigurable modules (PRMs). There is even a driver for the Linux kernel[13] that allows the partially reconfigurable regions (PRR) to be reconfigured with a simple command, such as cat module.bit > /dev/icap. Hence, this kind of usage is well developed and offers a high level of abstraction for the FPGA reconfiguration. Applications that interface with the ICAP directly, without abstractions, are harder to find. Here are a few: A scalable architecture for DCT (discrete cosine transform) computations[4], where the DCT and ME (motion estimation, the process of determing motion vectors that describe the transformation between images) are implemented as separate PRMs that are distributed to eight different PRRs, depending on which operation that is desired to prioritize. An attempt to reduce the FPGA device utilization of a sensor reading system for road safety by using dynamic partial reconfiguration[5]. Using partial reconfiguration in FPGAs to emulate switch-level faults[6]. The Metawire project, where the FPGA configuration circuitry is used as a NoC (network-on-chip)[12]. 9

24 10 Partial reconfiguration The Xilinx toolchain and partial reconfiguration Evaluating the (Xilinx) toolchain is somewhat problematic. In order to use the partial reconfiguration features, a special license is needed. To further complicate things, the license that is needed for version 9 of the Xilinx ISE synthesis tools is somewhat easy to acquire ( lounge access ), while it is much more difficult to get access to the license needed for version 11 ( restricted access ). ISE version 10 does not support PR at all (except for the difference based flow described in section 3.2.1). ISE 12 supports PR but requires a costly license. One gets the feeling that it is a catch-22; the tools are barely good enough for general usage which restrain people from using them, and the tools are lacking in usability and features perhaps because not that many people use them. ISE version 9 is used in this thesis, but most of the technical details should apply also for newer versions of the toolchain. What has changed mostly between the different releases is the flow used to design a system utilizing partial reconfiguration. ISE version 12 was released just before the completion of this thesis. It contains the ability to perform partial reconfigurations but requires a license. Even for university usage, the license is costly. Due to the price and limited time, it was unfortunately not possible to evaluate the PR features of ISE 12 in the scope of this thesis. 3.2 Technical details There are two approaches to partial reconfiguration in Xilinx devices; difference based and module based Difference based partial reconfiguration The difference based flow is feasible only for minor changes in the design (most often done in FPGA Editor). The generated bitstream contains only the bits that differ between the two designs. This flow does not need a special PR license; it is possible to generate these partial bitstreams by calling bitgen with the right switches, i.e: bitgen -g ActiveReconfig:Yes -r <orig.bit> <changed.ncd> <part.bit> which generates the partial bitstream part.bit containing the changes between the original bitstream orig.bit and the changed design changed.ncd. Using this flow for larger changes has not been attempted, but it is recommended for small changes only[15] Module based partial reconfiguration The module based flow is recommended for larger PRMs. A PRR is defined with a name and its coordinates (every PRR spans over a rectangular area in the FPGA logic, which is defined from the bottom left part to the upper right part) in the

25 3.2 Technical details 11 user constraints file. The bitstreams for the PRMs belonging to this PRR each contain the configuration data for the entire rectangular area, no matter how much logic in the area that is actually used by the particular PRM. Thus, the time required to configure a PRR is directly dependent on its area. Module based partial reconfiguration is what is used throughout this thesis. Figure 3.1 illustrates a design flow for module based partial reconfiguration. Static part RTL source code Constraints (UCF) xst NGC ngdbuild NGD map Partially reconfigurable modules RTL source code Constraints (UCF) xst NGC ngdbuild NGD map NCD (mapped) Constraints (PCF) NCD (mapped) Constraints (PCF) par par NCD (routed) static.used NCD (routed) bitgen bitgen Bitstream Partial bitstream Figure 3.1: Overview of a design flow from RTL code to bitstream using the Xilinx toolchain and several PRMs. The flow for the partially reconfigurable modules is repeated for each module. The static.used file contains routes that are used by the static design ICAP for partial reconfiguration Interface details The Virtex-4 ICAP interface is shown in figure 3.2. The write enable (we_i) and chip enable (ce_i) signals are active low. The busy signal (busy_o) is pulled high whenever the interface cannot receive data. During normal operation, the only time busy_o is pulled high is when changing read/write mode. The data bus width on the ICAP interface in different FPGA families varies, and is sometimes configurable. Widths of ports in some common FPGA families can be found in table 3.1. One important thing to keep in mind is that at least for Xilinx ISE (integrated software environment) versions up to (and including) 12.1[21], the place and route

26 12 Partial reconfiguration tool (par) does not consider setup timing requirements for the ICAP. Due to this, it is important not to have too short routed paths for the signals to ICAP in order to not violate the setup time constraint. This fact was the cause of many random errors during a long time for this project. There are two basic techniques to prevent this problem: one is to manually place the slices which produce the data to the ICAP input ports on a decent physical distance from the ICAP, another is to route the data wires to the ICAP through dummy lookup tables in order to increase the delay. clk_i en_i we_i data_i[31:0] ICAP data_o[31:0] busy_o Figure 3.2: Inputs and outputs of the Virtex-4 ICAP port in 32 bits mode Family Width (bits) Spartan-3 8 Spartan-6 16 Virtex-2 8 Virtex-4 8 or 32 Virtex-5 8, 16 or 32 Virtex-6 8, 16 or 32 Table 3.1: ICAP data port widths in different Xilinx FPGA families Clock frequency The Virtex-4 ICAP is specified to be able to run continuously at 100 MHz[20], however, as shown in [1], it is possible to overclock the ICAP port up to around 140 MHz by observing the busy signal from the ICAP interface. This yields a theoretical maximum throughput of configuration data of 560 Mbit/s (4 bytes per clock cycle). The paper also mentions the ability to clock the Virtex- 5 ICAP three times faster than specified; 300 MHz, yielding a throughput of 1200 Mbit/s. In order to confirm the correctness of the configurations, readback and comparison of the data were performed after configuration to guarantee a successful operation. Shortening the configuration time (and thus the overhead added with partial reconfiguration) is vital in many applications, especially when the PRMs are changed at a high rate. No ICAP overclocking has been attempted in this thesis.

27 3.2 Technical details 13 ICAP usage example Appendix C shows a simple example design that uses ICAP for performing readback of a configuration frame in a Virtex-4 device. The registers used for sending data to ICAP are placed via the UCF file, as explained in section In the icap_if module there is commented code that uses the other technique; routing the data through lookup tables in order to increase the delay Bus macros All information in this section is based on the Xilinx PR flow for ISE version 9, newer versions have not been considered due to the reasons pointed out in section All communication between the PRMs and the static parts of the device have to be done using bus macros[16]. Bus macros are used to lock the routing between the PRMs and the static portion. Only global signals (including clocks), VCC and GND can be used without bus macros. Each bus macro can handle an eight bit wide bus, which may optionally be registered. The placement of each bus macro has to be specified in the UCF file.

28

29 Chapter 4 Troubleshooting There are several options available for finding problems that generates unexpected results in a design. A large amount of troubleshooting have been required throughout the work with this thesis, mostly due to working with undocumented features, but also due to difficulties with simulating e.g. partial reconfiguration. Some techniques that have been useful for this particular project are presented here. 4.1 Simulation The obvious way of confirming a wanted behavior of a design is to perform a simulation. While simulating the design, all internal signals can be observed which is a great aid for finding and tracing the cause of any errors. Vital for efficient and useful simulations are well designed test benches, which provide stimulus to the design in order to simulate a realistic situation. Desired is to test all possible combinations of input data in order to consider every possible case of the design. This is however not feasible in most cases, since the amount of combinations usually is very high, and will result in a simulation time exceeding what is desirable. Instead, it is common to consider corner cases, which are sets of inputs that will test special cases in the design. The purpose is to make sure that the special cases are triggered only when they should. It might demand more of the designer to identify all corner cases, but it will save a large amount of time spent on the simulations. If simulations of a large design have to be performed over a longer period of time, it might be a good idea to use snapshots to decrease the probability of failures in the simulation. While using snapshots, the simulator will save the simulation state periodically. If the simulation hangs or ends due to some other reason, it is possible to resume it at another time. It is also possible to perform a faster simulation (for instance without visually displaying the waveforms), detect errors with a well written test bench, and then go back to the last snapshot where everything was working as expected, and then continue the simulation from that point, while observing the waveforms of the signals. 15

30 16 Troubleshooting Functional simulation The most common type of simulation is a functional simulation, also known as register transfer level (RTL) simulation. It is straightforward but will not take optimizations of the synthesis tools into consideration, neither will it consider the timing issues imposed by the routing. There are, however, more timing accurate simulation models available, of which two are presented below. These simulations will also show errors induced by the compiler Post synthesis simulation When performing a post synthesis simulation, the synthesized gate-level netlist will be used to model the system. No gate delays or propagation delays will be considered. Optimizations performed by the translate tool might change the appearance of the signals and the state machines, which can make it problematic to find and understand them in the simulator. For instance, when using the Xilinx toolchain, state machine variables are commonly Gray or one-hot encoded by the synthesis tool. This simulation type is good to find cases where the simulator and the synthesis tools behave differently. One case where the behavior may differ is when signal values other than 0 and 1 are used, such as X and Z. A typical command sequence for generating the required files and performing a post synthesis simulation using Verilog, the Xilinx tool chain and Modelsim is as follows: > xst -ifn design.scr -ofn design.ngc > netgen -w -ofmt verilog -sim design.ngc postsynth.v > vlog +acc postsynth.v > vopt +acc design glbl -L unisims_ver -o postsynth_opt > vsim postsynth_opt using the script file design.scr: run -ifn design.v -ifmt mixed -ofn design.ngc -ofmt NGC -p part -opt_mode Speed -opt_level 1 -top design Where design.v is the source code file, design.ngc is the NGC file containing the design and design is the top level name of the design Post place-and-route (PAR) simulation This is also known as timing simulation. It is a more complex simulation model, where the true timing delays and propagation delays of the hardware instances are considered. All hardware primitives and their interconnect are assembled in a single V or VHD file, and their timing information is placed in an SDF (standard delay format) file. It is more cumbersome to perform this kind of simulation, since there are a lot more signals available in the simulator than with the other simulation techniques, partly due to optimizations and partly due to the more primitive nature of this level of simulation. In order to trace the value a signal,

31 4.2 Logic analyzer 17 find the signal in the netlist file and see which other signals it depends on, find out their dependencies and repeat this process on the dependencies in an iterative fashion until the source that generates the unwanted result is found. This type of simulation is slower since also timing has to be considered. It is important not to violate the setup times for the circuitry while designing the test bench, since this might lead to unexpected results. For example, no signals should be changed at the rising edge of the clock: clk_i) begin data_i <= 1 b1; end should instead be written as (for instance): clk_i) begin #5 data_i <= 1 b1; end A command sequence to perform a timing simulation is similar to the example given in the end of chapter In order to get the correct simulation primitives, unisims_ver should be replaced with simprims_ver, and netgen should instead be run on the NCD (Native Circuit Description) file. 4.2 Logic analyzer One way of asserting the correct behavior of a circuit is to measure the signals while the device is running. This can be done either using an internal logic analyzer, such as Xilinx commercial ChipScope Pro (uses the boundary scan interface for measurements and JTAG for communication), or by physically connect an external logic analyzer Internal logic analyzer The internal logic analyzers are modules of varying size that are added to the design. The signals that are to be measured are connected to the logic analyzer and triggers are defined. One drawback with this method is that it occupies resources in the device, and that it most likely will result in changes in the routing of the design (hence it might impact timing issues). In some cases, it will also change the mapping of the design since it forces all parts of the measured signals to be used in their original form, which prevents some optimization to be done by the tools External logic analyzer An alternative is to use an external logic analyzer, which should have less impact on the design. The signals that are to be measured are routed to external pins,

32 18 Troubleshooting (a) Avnet Virtex-4 Evaluation Board with probes attached (b) Tektronix TLA721 Logic Analyzer Figure 4.1: Measurements with a logic analyzer and then probes from the logic analyzer are connected to these pins. The routing to the external pins can be done either via the user constraints file, or manually in for instance FPGA Editor after the design has been fully routed (as shown below). The latter is to be preferred, since it will not affect the routing of anything but interconnect. However, by adding more load (increasing the capacitance) on a signal net, it will get a slightly worse timing, which can affect the result in rare cases. Also, if the signals have been optimized in some manner during synthesis, they might not appear as expected in the placed and routed design. The most convenient way to add probes to a design using FPGA Editor is to use its command line equivalent, fpga_edline, in batch mode. It takes an SCR file and the NCD file as arguments and performs the actions described in the SCR file on the design. An example of an SCR file that routes four internal signals to external pins: probe script begin setattr main edit-mode read-write probe add mcu/io/uart/next_tx_o -pinname probe add mcu/controller/pc<0> -pinname probe add mcu/controller/pc<1> -pinname probe add mcu/controller/pc<2> -pinname save exit probe script end D13 E13 D14 F14 -usedpin -usedpin -usedpin -usedpin D13 E13 D14 F14 The command for applying the changes to the design can be invoked with > fpga_edline -p probes.scr system.ncd

33 4.3 Simulating PR designs Simulating PR designs It is currently not possible to simulate partial reconfigurability in a design. What is possible is to simulate each module separately (including the bus macros with the provided simulation primitives), but changing the active PRM while simulating is not supported. The ICAP simulation primitive is just a dumb module without any functionality whatsoever, which is understandable, considering it is a port into the FPGA configuration. However, supporting changing PRMs during simulation is an important feature in future simulators. It is an important step to get more designers to use partial reconfiguration in their designs. 4.4 Conclusion There are many options with varying properties available for debugging a design during development. Which one to use depends on several things, such as the type of problem and what has been attempted before. In general, it is best to start with a functional simulation and continue with the more complicated simulation techniques if the problem cannot be found. In some cases measurements with a logic analyzer may be needed, for instance when external hardware or ICAP are used. While developing in this thesis, all the methods described in this chapter have been used. Some use cases: Functional simulation The most common simulation method to assure that the design behaves as expected. Post synthesis simulation Used for instance to find differences in behavior between the synthesis tool and the simulator. One Verilog example where the result differed: if (x % 2) begin {...} which in a functional simulation worked as expected (perform the actions if x is odd), while the post synthesis simulation got different results. Post place-and-route simulation This technique has only been used to confirm that the final design behaves as wanted. Internal logic analyzer A technique used extensively to for instance monitor the inputs and outputs on the ICAP (which is impossible to simulate). It has been a major aid in the development of the BRAM debug tool.

34 20 Troubleshooting External logic analyzer When the behavior of a design changed when adding an internal logic analyzer to measure its signals (it suddenly started function as expected when being measured), measurements with an external logic analyzer helped finding the faults. One example is realizing that the setup times for the ICAP were violated.

35 Chapter 5 Motivation for a debug framework 5.1 Use cases In this report we have identified three use cases with varying availability of programming tools and interfaces (JTAG, serial port) where debugging an FPGA design might be desirable and possible: In the field consider the design deployed and that it suddenly starts to act unexpectedly. The person doing the debugging has probably not been involved in the design of the system, and does not have access to programming tools or JTAG. No tools available Serial port available In the field as the first case but with the availability of programming tools and possibly JTAG. Programming tools available JTAG (and possibly serial port) available During development of the design the person performing the debugging has most likely been involved of the development of the entire design. Programming tools available JTAG (and possibly serial port) available Design source code available When JTAG interface access is available, there are some solutions that can be used for debugging, such as the ChipScope logic analyzer. However, without JTAG, 21

36 22 Motivation for a debug framework the available options are very limited. We felt the need for debugging capabilities without the availability of programming tools and JTAG interface. On most FPGA boards there is a serial port available for communication to the outer world. Such a serial port is a rather easy task to add to a design since it only occupies two FPGA pins and the extra hardware needed for the physical interface is well standardized and cheap. Serial interfaces have been available on most computers for decades. Our idea was to exploit this possibility; to provide debugging capability with communication over a serial port, without requiring extra hardware or software. 5.2 Partial reconfiguration In order to save FPGA resources, the framework should exploit the partial reconfiguration capability of the devices. The system should provide several different modules each with a specific purpose for finding a specific type of fault. Only the module that is used at a certain time should occupy resources in the device, by being partially reconfigured.

37 Chapter 6 Debug framework idea and implementation 6.1 Introduction As an example application utilizing partial reconfiguration a debug framework was decided upon. The system consists of a range of different debug tools that are swapped in and out dynamically depending on choices made by the user. The idea is to provide debug capabilities on FPGA designs, with communication with the user over a serial interface. The debug system is added to the design to be tested (unit under test, UUT) using a methodology described in [2], which is further refined in this thesis. One aim is that it should be relatively straightforward to add additional partially reconfigurable debug modules to the system. This chapter presents the framework concept System overview The system is heavily based on the architecture used for the logic analyzer developed as part of the PyXDL library in [2], which is released under the GPL 1 (PyXDL) and MIT 2 (logic analyzer) licenses. Some changes to the architecture have been made in order to make it more suitable as a more general debug framework, namely: Making it possible to disable the MCU instruction execution (that is, adding an enable signal) Explained in section Extending the program memory with a write port usable by the I/O modules 1 Available at 2 Available at 23

38 24 Debug framework idea and implementation A design choice made in order to avoid using additional BRAMs in the BRAM debug tool (a module which is described in detail in chapter 7) As described more thorough in section 3.2.4, the signals between the static and dynamic modules have to be routed through bus macros. Bus macros have been added for all module crossing signals in each of the PRMs. UART RS-232 Microcontroller I/O ICAP Program memory PRR Logic analyzer BRAM debug tool Figure 6.1: System overview of the debug framework Debug modules Two modules will be included in order to show the framework concept. The first is a logic analyzer (presented in [2]), with a 32 bit wide bus for data capture and trigger. The second is a tool used to read out the content of BRAMs developed in the scope of this thesis (presented in detail in chapter 7). These tools are implemented as PRMs that share a partially reconfigurable region (PRR), as shown in figure 6.1. Below are some ideas for additional modules that are presented briefly. The resource usage estimations for the different modules are based on the actual numbers from the BRAM debug tool and the logic analyzer, but they should be interpreted as rough estimates. The LUT numbers are based on 4-input LUTs. LUT and flip-flop altering capability A module that allows the content of LUTs and flip-flops to be observed and altered easily. The module would read and alter the values by using the ICAP. The state machine would be somewhat similar to the state machine used in the BRAM debug tool; the same commands are used for reading out this kind of data. The time and area consuming bit transformation step can be removed, while the editing capability would add another state machine for writing the data plus intermediate storage for the configuration frame data (the entire frame has to be read out, the changes should be made and then the entire frame should be written back). Estimated resource usage and motivation: Readout 160 LUTs, 1 BRAM

39 6.1 Introduction 25 The BRAM debug tool occupies around 160 LUTs without the bit transformation Editing 30 LUTs A simple edit function that lets the user edit a row in the BRAM Writing 200 LUTs This state machine would be a bit more complicated than the one for data readout. In total 390 LUTs and one BRAM. Configuration readback A configuration readback module to read back a part of (starting from a specific address) or the entire FPGA configuration to a host computer. Considering that the communication with the user is done via the serial interface, the data has to be buffered and read out in sets because the ICAP delivers data at a much higher rate than what can be transferred over the serial port. It is probably best to read out rather small amounts of data at a time, enough to fill one or perhaps two BRAMs. If the BRAM data in the bitstream is desired some other buffer solution has to be used, such as distributed RAM. However, since the BRAM data already can be read out using the BRAM debug tool, this module should focus on logic and interconnect. Estimated resource usage and motivation: Readout 160 LUTs, 1 BRAM The BRAM debug tool occupies around 160 LUTs without the bit transformation Other logic 30 LUTs Just some basic logic to handle for instance the frame addressing and MCU interfacing In total 190 LUTs and one BRAM. Statistics module A module that collects statistics on signals, i.e., the amount of time the signals have a certain value. In order to be able to do this efficiently for several signals at once, a system with hierarchical counters is suggested. Each counter counts the number of matches for one specific signal/bus. As an example, consider a Wishbone bus with two masters and eight slaves. It is desired to know how much each device is used, and in what way (reading/writing etc). Since several devices are active at a time, one memory for storage is not enough (each memory update is a readout followed by a write operation). Our

40 26 Debug framework idea and implementation idea is to have a small counter for each signal that is to be measured. Periodically, the values of all these local counters are collected using a simple round-robin scheduling scheme and saved in a main memory dedicated for storage of signal statistics. It must be possible to get statistics for each device individually. For each device, two counters are needed; one for counting write operations and one for read operations. In total 20 counters. The local counters can be read out serially in order to save logic, which means that it takes as many clock cycles to read out one counter as the counter is wide (in bits). Thus, each local counter can be read and saved in the main memory every 20 w clock cycles, where w is the counter width. The counter minimum width is dependent on the maximum value the counter may have, which is dependent on how often it will be read to the main memory. This gives the following inequality: 2 w 1 > 20 w (6.1) Which holds for w > 7. Hence, w = 8 should be the optimal choice provided that the memory is dedicated and only used for this purpose. The main memory needs to be able to address at least 20 addresses, and contain an adder to add the current memory value with the local counter value. It is possible to use the program memory for this if an extra read port is added, and the data width is adjusted accordingly. Estimated resource usage and motivation: Local counters 20 8 LUTs One eight bit counter occupies eight LUTs and there are twenty counters needed Round-robin data collector 90 The data will be transferred serially to the data collector which should be resource efficient. Other than the data transfer, the data collector basically only reads out one value from the memory, adds the stored value with the current counter value and saves it on the same address. In total 250 LUTs. Bus interface A bus interface for one or several bus standards, such as Wishbone, PLB (processor local bus) or OPB (on-chip peripheral bus). It should be possible to snoop on all interesting signals on a slave device, for instance data in, data out, address, write enable and strobe. Considering a 32 bit data bus and a 16 bit address bus, a total of 82 signals should be taken into consideration. It should be possible to write and read words of data to and from the bus. Estimated resource usage and motivation: Triggering and data storage 140 LUTs

41 6.2 Implementation details signal values to be saved for trigger and data (the same location can be used) and logic for comparison Other logic 30 LUTs Control logic In total 170 LUTs. 6.2 Implementation details Microcontroller This is a simple 8-bit microcontroller unit (MCU) developed by Andreas Ehliar. It is designed for being small (use small amount of resources when implemented in an FPGA or in an ASIC) and for relatively high clock frequencies (in order to not be the limiting part of the whole merged design). It is prepared for pipelining. However, no pipelining is currently used since the extra performance it potentially would offer is not currently needed, and it would require more hardware resources for the implementation. The MCU is basically only handling data transfers to and from I/O devices and used to manage the text based user interface. For the programmer, 16 general purpose registers, each 8 bits wide, are available for data. Programming The MCU implements around ten general instructions plus five additional branch related instructions. The MCU is programmed using a simple assembly language that is assembled using an assembler written in Perl. The assembler handles labels and outputs the machine code (the program memory) as the contents of a BRAM in a Verilog code file Program memory The program memory contains the program, constants used and possibly also data from the BRAM debug tool. It contains 1024 rows, each 16 bits wide. The MCU only has a read port, thus no data can be written to the memory by the programmer during execution. There is a write-only port available for the I/O that is currently used by the BRAM debug tool, using the reserved addressing space ranging from 0x3ef to 0x3fe UART The universal asynchronous receiver/transmitter (UART) handles the serial interface communication. It takes bytes as inputs and sends them serially, and similarly, it retrieves serially and outputs bytes to the rest of the system. In this system it is configured for a baud rate of 115,200 bit/s, with an 8-N-1 configuration (eight data bits, no parity bit, one stop bit).

42 28 Debug framework idea and implementation ICAP The ICAP module is described in section 7.3.3, with the addition that the data to this module when used in the debug framework is multiplexed in the I/O module to allow several submodules to use it PRR This can be seen as an abstract module that houses the PRMs. The PRMs defines different data interfaces whose signals are routed through bus macros. 6.3 Design flow The original idea was to add the system to an already placed and routed design. However, with newer versions of the tools, the ability to place and route a partially unplaced design has changed. Since the debug framework currently is designed for (and with) ISE 9, it is not much of an issue here. The modified design flow for synthesizing and merging the designs is illustrated in figure XDL The Xilinx toolchain has the ability to output ASCII translations of mapped, partially routed or fully routed designs in an XDL file. It is an ASCII variant of the binary, undocumented NCD file format. Designs can be converted between these formats by using the xdl command line tool. For instance, it is possible to convert a fully routed design into XDL, make some changes, convert it back to NCD and then only be required to route the unrouted parts (using par in guided mode). This ability creates possibilities to alter designs in a late stage of the design process without having to go through the whole (often lengthy) procedure of synthesizing and mapping. PyXDL PyXDL 3 is a Python library developed to simplify the process of modifying designs via the XDL file. Its features include modifications on entire networks, design merging and much more. It is used for merging the UUT and the debug tool by performing the following operations: 1. Import the design to be debugged (UUT) and the design of the debug tool (DT) 2. Find the clock network for the UUT 3. Unplace the DT 4. Remove any unused DCMs (digital clock managers) from the DT 3 Available at ehliar/

43 6.3 Design flow 29 RTL source code for UUT Synthesizer (xst) NGC ngdbuild NGD map Constraints (UCF) RTL code for the static part of the debug framework Synthesizer (xst) NGC ngdbuild NGD map Constraints (PCF) Constraints (PCF) NCD (mapped) par NCD (routed) xdl XDL (routed) PyXDL design merger XDL (partially routed) xdl NCD (partially routed) par NCD (mapped) xdl XDL (mapped) BRAM name & address extraction RTL code for the PRMs Synthesizer (xst) NGC ngdbuild NGD map NCD (routed) bitgen Bitstream Constraints (PCF) NCD (mapped) par NCD (routed) bitgen Partial bitstreams Figure 6.2: Overview of the modified design flow using the Xilinx toolchain, based on the flow described in [2]. The steps in the upper right red area is for synthesizing the static parts of the debug framework while the lower right blue area is for synthesizing the debug framework modules. The BRAM name & address extraction step is only required if the BRAM debug tool PRM is desired.

44 30 Debug framework idea and implementation 5. Remove the clock network from the DT 6. Create a unique prefix for the DT to avoid name collisions 7. Merge the designs 8. Add the clock network to the DT 9. Place and route the merged design 6.4 Partial reconfiguration Storage of PRM bitstreams The PRM bitstreams can be stored either on a computer connected to the FPGA, in an off-chip memory located more closely to the FPGA, or even in a memory located inside the FPGA. There are pros and cons with each solution: Computer storage, transfer via JTAG + Does not use resources in the FPGA Requires a computer with the needed tools connected to the board when reconfiguring Needs human intervention for the reconfiguration or special developed software that handles the reconfiguration Computer storage, transfer via serial port + Does not use resources in the FPGA Requires a secure protocol that can handle the bit errors that may occur during transfer Requires a computer with special software for the transfer Slow Needs human intervention for the reconfiguration or special developed software that handles the reconfiguration Off-chip memory close to the FPGA + Does not use resources in the FPGA Does use hardware resources that may be needed for other modules Might not be available in all cases we cannot expect to be able to use such a memory Might not be fast enough and/or may negatively affect the critical path Memory in the FPGA (BRAM) + Fast

45 6.5 Resource usage 31 Uses a lot of hardware resources on the FPGA May be problematic depending on where the BRAMs are located, and whether there are BRAM configuration data in the PRMs (this issue is more thoroughly discussed in section 7.1.1) Another option was also reviewed; using the primitives USR_ACCESS_VIRTEX4 and STARTUP_VIRTEX4 to use the static bitstream as storage location for the partial bitstreams. It is possible to add additional data in the bitstream which can be read out from inside the FPGA using the USR_ACCESS_VIRTEX4 primitive, with the help of STARTUP_VIRTEX4. The data can be added to the bitstream with a Perl script presented in [14], where caches for a PowerPC processor are loaded with data upon start of the device. However, the data added to the bitstream can only be read out right after the device has been started, which unfortunately makes it useless for this case. It would be desirable to be able to use the configuration memory for storing partial bitstreams but at the moment it does not seems to be possible. 6.5 Resource usage Module LUTs BRAMs Bare system without any module BRAM debug tool Logic analyzer (32 bit) The logic resource usage for the two implemented modules are very similar, which potentially should lower the amount of unused logic in the PRR.

46

47 Chapter 7 Block RAM debug tool 7.1 Description Since the development of this module occupied the majority of the time used for the thesis, and because it was developed entirely for the thesis, it is presented in detail here. The purpose of this tool is to simplify the process of finding faults in other designs by allowing the designer to see the content of the BRAMs used in those designs. The BRAM debug tool will be released freely available independent from the rest of the system, in a more simple variant not using partial reconfiguration Block RAMs and ICAP As stated in [18], using BRAMs while reading their data back via ICAP might lead to problems. This behavior was slightly investigated; a BRAM was setup with both ports connected. The contents of that BRAM were then read back via ICAP simultaneously as both ports were writing (to different addresses). Further the entire BRAM contents were read out (with ICAP inactive), to see which of the write operations that had been successful. The result was somewhat surprising; during some cycles the first port was blocked, during other cycles the second port was blocked. No real pattern was found for which memory port ICAP uses for readback. As pointed out in [12], the readback of a certain BRAM even affects the usage of BRAMs in other areas of the chip (BRAMs with completely different addresses and locations), in an undocumented manner. This shortcoming leads to some constraints in the design of this tool: No parts can be allowed to use any BRAM during readback via ICAP The data read back using ICAP has to be stored in some alternative location (that is, not in a BRAM) The first constraint is solved by disabling the MCU temporarily during readback, which has to be done since its program memory is located in a BRAM. 33

48 34 Block RAM debug tool The second constraint is handled by temporarily storing the data in 32 parallel distributed RAMs. 7.2 Block RAM data in the bitstream Addressing scheme Block RAMs are usually referenced to using X and Y coordinates, such as RAMB16_XnYm, where n denotes the physical column in the FPGA where the BRAM is located, and m determines the row. The XC4VLX25 device has three columns containing 24 BRAMs each, with a total storage capacity of = 1296 Kbits. The frame address register (FAR) is used to specify which bitstream address that it is desired to begin reading from (or to). In order to be able to send the correct address to the FAR, the X,Y coordinates need to be translated as specified in [18]. The smallest addressable segment of the configuration memory space is a frame, i.e bit words. Description of each of the 23 bits in the Virtex-4 FAR: Top/bottom {}}{ b 22 b 21 b 20 b 19 }{{} Block type Row address {}}{ b 18 b 17 b 16 b 15 b 14 b 13 b 12 b 11 b 10 b 9 b 8 b 7 b 6 }{{} Column address Minor address {}}{ b 5 b 4 b 3 b 2 b 1 b 0 The top/bottom bit should be set to 0 if the BRAM is located in the top half of the FPGA (the Y coordinate is larger than or equal to half the device specific number of BRAM rows). For example, consider the XC4VLX25 device with 24 BRAM rows (indices 0-23), where the BRAMs with an Y coordinate between 12 and 23 is located in the top half. For BRAM contents, which is the desired type in this case, the block type should be set to 010. The row address specifies the distance from the center of the device to the current BRAM, as shown in figure 7.1. The column address starts at 0 at the leftmost column and is directly taken from the X coordinate. The minor address in the case with BRAMs denotes which segment that is desired. More information about the data segments and an addressing example involving minor addressing can be found in section See figure 7.1 for an illustration of the BRAM addressing scheme in the XC4VLX25.

49 7.2 Block RAM data in the bitstream 35 BRAM16_X0Y23 BRAM16_X1Y23 BRAM16_X2Y23 BRAM16_X0Y22 BRAM16_X0Y21 0x BRAM16_X1Y22 BRAM16_X1Y21 0x BRAM16_X2Y22 BRAM16_X2Y21 0x BRAM16_X0Y20 BRAM16_X1Y20 BRAM16_X2Y20 BRAM16_X0Y19 BRAM16_X1Y19 BRAM16_X2Y19 BRAM16_X0Y18 BRAM16_X0Y17 0x BRAM16_X1Y18 BRAM16_X1Y17 0x BRAM16_X2Y18 BRAM16_X2Y17 0x BRAM16_X0Y16 BRAM16_X1Y16 BRAM16_X2Y16 BRAM16_X0Y15 BRAM16_X1Y15 BRAM16_X2Y15 BRAM16_X0Y14 BRAM16_X0Y13 0x BRAM16_X1Y14 BRAM16_X1Y13 0x BRAM16_X2Y14 BRAM16_X2Y13 0x BRAM16_X0Y12 BRAM16_X1Y12 BRAM16_X2Y12 Center BRAM16_X0Y11 BRAM16_X1Y11 BRAM16_X2Y11 BRAM16_X0Y10 BRAM16_X0Y9 0x BRAM16_X1Y10 BRAM16_X1Y9 0x BRAM16_X2Y10 BRAM16_X2Y9 0x BRAM16_X0Y8 BRAM16_X1Y8 BRAM16_X2Y8 BRAM16_X0Y7 BRAM16_X1Y7 BRAM16_X2Y7 BRAM16_X0Y6 BRAM16_X0Y5 0x BRAM16_X1Y6 BRAM16_X1Y5 0x BRAM16_X2Y6 BRAM16_X2Y5 0x BRAM16_X0Y4 BRAM16_X1Y4 BRAM16_X2Y4 BRAM16_X0Y3 BRAM16_X1Y3 BRAM16_X2Y3 BRAM16_X0Y2 BRAM16_X0Y1 0x BRAM16_X1Y2 BRAM16_X1Y1 0x BRAM16_X2Y2 BRAM16_X2Y1 0x BRAM16_X0Y0 BRAM16_X1Y0 BRAM16_X2Y0 Figure 7.1: Location of BRAMs in the XC4VLX25 device, along with their names and bitstream addresses for their first data segments BRAM data segments Four BRAMs share the same address in the bitstream for the memory content. Their data is grouped into smaller segments each consisting of 256 bits of data and 64 parity bits. How these segments are located is illustrated in figure 7.2. The situation is not as trivial as one might expect (data from each BRAM is gathered in consecutive rows). Instead, the data from one BRAM is spread out and mixed up with data from all the other BRAMs sharing the same address.

50 36 Block RAM debug tool RAMB16_X2Y11 Segment 0, Offset 0 RAMB16_X2Y10 Segment 0, Offset 10 RAMB16_X2Y9 Segment 0, Offset 21 RAMB16_X2Y8 Segment 0, Offset 31 RAMB16_X2Y11 Segment 1, Offset 0 RAMB16_X2Y10 Segment 1, Offset 10 RAMB16_X2Y9 Segment 1, Offset 21 RAMB16_X2Y8 Segment 1, Offset 31 RAMB16_X2Y11 Segment 2, Offset 0 RAMB16_X2Y10 Segment 2, Offset 10 RAMB16_X2Y9 Segment 2, Offset 21 0x x x Figure 7.2: Illustration of the BRAM data segment offset and ordering for a group of four BRAMs in the bitstream for XC4VLX Bitstream offset Depending on the location of the desired BRAM within its addressing group, an offset has to be added in order to read out the correct data from the bitstream. There is also blank line between the second and the third segment on each address, which requires an extra offset to be used for the data located in the third and fourth data segment, as can be seen in figure Mapping of Block RAMs While reading the contents out of a BRAM using the ICAP interface, the data bits are transformed in an undocumented way. What is actually read through the ICAP port is parts of the bitstream, hence the mapping is the same as in the bitstream used when programming the FPGA. Arrays showing the location of all bits can be found in appendix A.1 for the upper half of the device and appendix A.2 for the lower half. The bit locations in the lower part are the same as in the upper part, but the entire matrix is flipped and mirrored. In order to have any use of this data, the mapping procedure had to be determined. This was done using information in [10], where the mapping for Virtex-II devices is analyzed,

51 7.3 Implementation 37 and by studying the LL (logical allocation) file generated by the bitgen bitstream generator. The mapping procedure in appendix B describes how to find a particular bit c j in a BRAM from a 320 bit BRAM content segment in the bitstream for a Virtex-4 device. 7.3 Implementation This section primarily considers the system when independent, i.e., not when used as a PRM in the debug framework. Figure 7.3 shows an overview of the system. The different modules are not functionally different from the debug framework, only the hierarchy has changed slightly. Of course, everything related to partial reconfiguration (such as bus macros) have been removed in order to save resources, ease development and minimize the critical path. Apart from these hardware modules, scripts are used in the flow to generate data that is needed for the tool to operate. UART RS-232 Microcontroller I/O ICAP controller Program memory ICAP interface Figure 7.3: System overview of the BRAM debug tool, when used independently Design flow Figure 7.4 illustrates the design flow. The UUT and the BRAM debug tool designs are merged with PyXDL after they have been mapped. After being placed and routed, the merged design is converted to XDL. The name, location and width of each BRAM are determined and the program memory is updated using a script developed for this tool. The XDL design is then converted back to NCD, and the bitstream is generated. This is different from the flow described in section 6.3. The reason is that with ISE versions later than 9, it is not possible to place unplaced components with par, which is needed in order to follow the debug framework design flow. Unfortunately, this means that the entire implementation process probably will take longer time. Instead of as before; merge the placed and routed UUT design in XDL format with the unplaced BRAM debug tool in XDL format, now the two designs are merged before being placed and routed. Thus, unfortunately, the

52 38 Block RAM debug tool placement and routing of the UUT might change when the BRAM debug tool is added. It is likely possible to merge the designs in the ngdbuild step, and then use SmartGuide (an ISE feature to shorten the execution time of map and par by using a previous design as guide) to more quickly place and route the entire merged design without changing the placement of the UUT design. Due to time constraints, it has not been investigated further. RTL source code for UUT Constraints (UCF) RTL code for the BRAM debug tool Constraints (UCF) Synthesizer (xst) Synthesizer (xst) NGC NGC ngdbuild ngdbuild NGD NGD map map NCD (mapped) Constraints (PCF) NCD (mapped) Constraints (PCF) xdl xdl XDL (mapped) XDL (mapped) PyXDL design merger XDL (merged) Constraints (PCF) BRAM name & address extraction xdl XDL (updated) NCD (merged) xdl par NCD (routed) NCD (routed) bitgen xdl Bitstream XDL (routed) Figure 7.4: Overview of the design flow for the BRAM debug tool using the Xilinx toolchain. The steps in the blue area are for synthesizing the BRAM debug tool.

53 7.3 Implementation ICAP controller The following code snippet shows the input and output signals of the ICAP controller module: module icap_ctrl( input wire clk_i, input wire rst_i, input wire readstrobe_i, input wire [1:0] offset_i, input wire [22:0] icap_addr_i, output reg [15:0] pm_data_o, output wire [3:0] pm_w_addr_o, output reg pm_we_o, output reg ready_o, output reg data_ready_o, output reg mcu_en_o ); Below are a number of vital steps in the ICAP controller described. BRAM readout The ICAP controller consists of a state machine that sends commands to the ICAP interface. It is designed to read out ten 32 bit words of data from the bitstream starting from the FAR address on the input signal icap_addr_i. As specified in section 2.2, the smallest addressable segment consists of bit words. In order to minimize the hardware needed for intermediate data storage, only 320 bits (256 data bits and 64 parity bits) of data are read at a time, which corresponds to ten words in the bitstream. These ten words are fetched out of the 41 read words on an offset specified by offset_i (a two bit wide bus its four possible values corresponds to an offset of 0, 10, 21 or 31 words as described in section 7.2.3). The data is stored in 32 parallel distributed RAMs. During the readout, the MCU is temporarily disabled via the mcu_en_o signal due to the reason outlined in section The data readout starts when readstrobe_i is 1 (which it should be exactly one clock cycle). The ICAP control module in appendix C is a simplified version of the ICAP controller that reads out an entire configuration frame. The first time it is run it executes the ICAP commands twice; this is because the ICAP returns the data with a different offset the first time a readback is performed after startup. The reason for this behavior is unknown. When looking at the software examples for Xilinx ICAP implementation, the behavior is dealt with in almost the same way; by always reading out the data twice. Bit mapping Two different approaches for performing the bit transformation described in section have been investigated: one based on the mapping procedure in appendix B and one utilizing pre-calculated mapping values stored in a BRAM. When comparing the resource usage for these solutions, it was clear that the approach based on the mapping procedure was more feasible. It provided a solution without requiring

54 40 Block RAM debug tool a BRAM while at the same occupying less logic (slices) than the BRAM based alternative. The maximum clock frequency is slightly lower with this solution, but since this system will operate independent of the UUT, the clock frequency is not very important. Data storage Once the data is correctly transformed, it is saved in the program memory (a design choice made to save resources and greatly simplify the assembly program) and the ready signals are pulled high ICAP interface This module is basically just a wrapper for the ICAP port. All data is registered and the flip-flops for the data to the ICAP interface are placed in slices located far enough from the ICAP interface due to the reasons described in section It is identical to the ICAP interface module in appendix C The block RAM name & address extraction script In order to set the correct memory data (name and address for each BRAM in the UUT) for the user interface (which can be seen in figure 7.5), the UUT design has to be analyzed. This is done by parsing the placed and routed XDL file for the UUT and generating memory contents that later are used by the microcontroller unit for drawing the menu to the user. This procedure is done using a Python script that reads the XDL file, and inserts the data regarding the BRAMs into a file containing the program memory for the MCU. The procedure in details: Parse the UUT XDL file using PyXDL Open the file containing the assembly code for the MCU Change four values (marked with AUTOGENERATED) that are used while drawing the menu (specifies menu width etc.) Iterate over all BRAMs found in the design, and for each BRAM: Calculate the bitstream address, as described in section Find the distance from the device center (for offset calculation) Save the width of the BRAM, the distance from the center, the bitstream address, its XnYm address and its name into an array Print the BRAM s name into the assembly code file (used for drawing the menu) Iterate through the previously created array with BRAM data and print the data into the assembly code Save the assembly code file

55 7.4 User manual 41 Resource usage The resource distribution is as follows: ########################################################################### # Name # SLICEL # SLICEM # RAMB16 # IOB # DSP48 # ICAP # ########################################################################### # / # 439 # 25 # 1 # 21 # 0 # 1 # # /mcu/ # 435 # 24 # 1 # 0 # 0 # 1 # # /mcu/io/ # 295 # 20 # 0 # 0 # 0 # 1 # # /mcu/io/icap/ # 211 # 20 # 0 # 0 # 0 # 1 # # /mcu/io/uart/ # 79 # 0 # 0 # 0 # 0 # 0 # # /mcu/controller/ # 107 # 4 # 0 # 0 # 0 # 0 # # /mcu/program_memory/ # 3 # 0 # 1 # 0 # 0 # 0 # ########################################################################### In total input LUTs are used. The maximum clock frequency with a 50 MHz TIMESPEC constraint is 140 MHz on an XC4VLX Other constraints have not been investigated since the maximum frequency already by far exceeds what is required to run the system. The limiting factor (regarding the clock frequency) in the system is the serial port, and it is possible to run the system two orders of magnitude slower than its current speed and still satisfy the requirements of the serial port. 7.4 User manual This user manual describes the tool when used independently, without partial reconfiguration Adding the tool to a design This flow is based on figure Synthesize the UUT up to the map stage 2. Convert its NCD design representation to the XDL format: > xdl -ncd2xdl design.ncd 3. Synthesize the BRAM debug tool up to the map stage 4. Convert its NCD representation into XDL 5. Merge the designs using PyXDL 6. Convert the merged XDL design into NCD 7. Place and route the merged design 8. Convert the placed and routed design into XDL 9. Run the BRAM name & address extraction script on the placed and routed XDL file to update the assembly code located in a program memory in the XDL file: > python parser/parser.py merge_par.xdl > merge_par_updated.xdl

56 42 Block RAM debug tool 10. Convert the updated XDL file back to NCD: > xdl -xdl2ncd merge_par_updated.ncd 11. Generate the bitstream: > bitgen merge_par_updated.ncd 12. Program the device Using the tool Figure 7.5: Screenshot of the tool while reading out the 16 bit wide MCU program memory. The last line is generated by minicom and is thus not part of the system. Connect a serial cable between the device and the computer, and start a terminal emulator (e.g. minicom on Unix-like systems, or HyperTerminal in Windows). Make sure that the serial port settings are set to 115,200 bit/s, with an 8-N-1 configuration (eight data bits, no parity bit, one stop bit). When started, the system will present a user interface similar to the upper part of figure 7.5. It prints a menu with the names of all BRAMs in the UUT. The user can select and print the entire contents of a BRAM by selecting it in the menu using the keys described in table 7.1. The > character indicates the currently chosen BRAM. Below the menu the X and Y coordinates of the chosen BRAM is printed. When the space key is pressed, the contents of the chosen BRAM will be printed. 256 bits are printed on each

57 7.4 User manual 43 Key w s a d Space Action Move up Move down Move left Move right Select Table 7.1: Key bindings for used for interaction with the user interface row, grouped depending on the width of the BRAM. 16 rows are printed at a time before the user has to press a key to continue the printing. In total 64 rows are printed for each BRAM. Figure 7.5 shows the first 16 rows of data from the program memory. Note that the data is printed in groups of 16 bits in this case, due to the width of the memory bus.

58

59 Chapter 8 Results 8.1 Partial reconfiguration One purpose with the thesis was to investigate the current state of partial reconfiguration. One major obstacle to this evaluation is the complicated license and version situation which is mentioned in section Since there are two newer releases of the Xilinx toolchain with PR support than the one investigated in this thesis, it is hard to claim that these results are generally valid. The results apply to a product released in 2007 that is about to reach its end-of-life at the time of writing. It would of course have been preferred to use a more recent version of the toolchain but due to the license issues it was not possible. However, since a lot of time has been spent using the PR features, and due to some time-consuming faults caused by using the PR flow, some reflections on the tools will be presented here CRC errors With some designs, in a more or less random fashion, the bitgen tool generates faulty bitstreams for the PRMs. It is a problem that is repeatable, i.e., it does not matter how many times the bitstream is generated, some designs just end up with CRC errors in their bitstreams. These errors are discovered first when the FPGA is programmed with a faulty bitstream (the FPGA performs a CRC check during configuration), most likely with a very odd behavior as result. In order to lower the probability of transmission errors, the JTAG clock frequency was decreased. However, the errors were still occuring. There must be some fault in bitgen that causes this error, no correlation between a certain property of a design and this issue could be found. At least, the bitstream generator should perform an internal CRC check on the design before writing it to disk in order to prevent errors like this. 45

60 46 Results Placement/routing issues During the development of the PRMs, some unexpected errors were experienced. Some signals that were routed through bus macros got incorrect values in some cases. The signals were measured with an external logic analyzer, and with some designs, they did not have the correct values. All bus macro signals were registered in the bus macros, and according to FPGA editor, the timing was not an issue (to assert that timing was not a problem, one design was tested running at half the normal clock frequency). The behavior was seemingly random; when small, non-critical changes were made to a design, it could suddenly begin to work (or fail). If the design was synthesized as a static module (while still using the bus macros), it always functioned as expected. Once the design was synthesized as a PRM, it could start to behave unexpectedly. One indication to that this is a placement/routing related issue is that at one time during development of a PRM, the name of a module was changed, which was enough to break its functionality. This is most likely caused by the map or par tools, which somehow reordered the placement/routing order of the primitives due to the name change (based on some internal representation of the design). The cause of these errors is still not known. 8.2 Debug framework The debug framework development was affected by, among others, the issues mentioned in section 8.1 (which was experienced while developing the debug framework). Since both the issue with CRC errors and the placement/routing problems were so unpredictable, it was hard to develop efficiently. We believe that the debug framework concept is a good idea, but with the ISE version used here, it is not feasible to use the PR features of it due to all the problems encountered. If a more recent ISE version with PR support is available, it would be interesting to see if it suffers from the same issues as the version used here. In general, if a newer PR supporting tool is not available, it is a better idea to use the debug modules one by one, without using PR. This would solve the CRC errors, placement/routing problems and the issues with the planned flow described below. Further it would not require any of the PR licenses and would work with at least ISE version 9 and newer. However, in order to partially configure an FPGA, a PR license is not needed. Thus, it is needed only if the design is modified and thereby needs to be resynthesized XDL As described in section 6.3.1, the XDL format is supposed to be used to merge the UUT design and the debug framework. Rather late in the development process, it became clear that the i_PR14 version of xdl does not support XDL conversion of PR based designs. In order to be able to merge the designs as planned, this is a required feature. Whether support for this exists in more recent versions of ISE is unknown.

61 8.2 Debug framework 47 It might be possible to merge designs before ngdbuild using the EDF files. However, it has not been attempted and whether it works with PR or not is unknown. The only way to currently solve this problem is to synthesize the two designs simultaneously (which unfortunately means that the time for adding the debug framework to a design increases, and it will most likely change the placement and routing of the UUT). There is no other feasible way to merge designs other than using the XDL technique Placement collisions In section 3.2.2, it is stated that the PRRs are defined (placement and dimensions) in the user constraints file. This might impose some problems to the described design flow in section 6.3; if the (for the PRRs) specified logic is occupied by the UUT, the merge process would fail. The same issues apply to the placement of the bus macros. This issue can be solved by synthesizing the designs simultaneously (that is, instantiate them in a common top design), or by preparing the UUT with PROHIBIT placement constraints that leaves a certain area unused, that later can be used by the debug framework. It might also be possible to unplace the problematic UUT logic (all unplaced logic will be re-placed by the par tool later in the process). Unplacing logic is however not very easily done, and it would potentially affect the behavior of the UUT design Logic analyzer Consider the idea discussed in section This is a great example of when partial reconfigurable modules can help bring down resource usage. Routing 128 signals through bus macros consumes 256 LUTs, which is very low compared to 1088 LUTs (+reconfiguration logic) that the proposed solution consumes. Compared to the naive solution, which needs more than 16 times as many LUTs as the bus macros, the difference is even more extreme. The drawbacks are that every wanted logic analyzer configuration has to be synthesized separately (but as mentioned, it can be scripted), and that the FPGA needs to be reconfigured each time it is desired to change which signals to measure. This is an example that really shows some of the advantages with partial reconfiguration Feasibility of the framework This section will compare the presented framework utilizing partial reconfiguration with a fully static system, with no PR. There are many aspects that should be taken into consideration in order to get a fair comparison. At the time of writing, it is a bit hard to justify the system due to all the problems with partial reconfiguration and ISE version 9. However, this section aims to compare the framework

62 48 Results concept as presented in chapter 6, not the actual resulting system. The comparison should give an indication whether it is feasible to attempt an implementation of the framework using a newer version of the toolchain or not. First, some comments on the two major problems regarding XDL and placement collisions affecting the flow as mentioned above: XDL This issue is caused by a shortcoming of the tools used. Since it is not known if there is support for converting designs to the XDL format with newer tools, it is hard to say whether the issue would remain an obstacle. This issue is however a major setback for the flow, an issue that would not exist with a system not utilizing PR. Placement collisions This issue would most likely remain with newer versions of the toolchain. There are three feasible solutions to circumvent it: Synthesize the designs simultaneously Use the PROHIBIT placement constraint Use a non-pr variant of the framework Potential resource savings In order to save hardware by using PR, it is obvious that at least two PRMs are needed to compensate for the overhead imposed by the PR technique, such as bus macros and placement constraints. This section aims to give an answer to how large this overhead really is, and how many and which modules that are needed to motivate the usage of PR. Since only two modules have been developed, the ideas for additional module implementations presented in section are considered as well. There are two questions that need answers in order to be able to give a good answer to whether it is worth using PR or not in a certain situation: 1. Storage location of the PRM bitstreams 2. Which modules that are desired to use in a specific case The storage location might have a huge impact on the total resource utilization depending on which solution that is chosen. The different viable options are discussed in section The three most reasonable storage solutions will be considered here: On a computer connected to the device, data transfer via JTAG On a computer connected to the device, data transfer via serial port In BRAMs in the FPGA

63 8.2 Debug framework 49 The first option gives an advantage to the PR solution, but comes with the drawback that it needs a computer connected when changing the active module. One of the motivations for developing the debug framework was to provide debug possibilities when JTAG is not available, which is a bit contradictory if the bitstreams are to be transferred via JTAG. The second option needs further development of both software and hardware, but does not require JTAG. The third option comes with the obvious disadvantage (for PR) that (potentially a lot) of memory resources will be used for bitstream storage. However, it is impossible to give an answer to what is best in all cases. It is usually a trade-off between different aspects, such as decreasing flexibility by storing the bitstreams on the computer with decreased hardware usage as a result, or get logic savings by utilizing PR but sacrifice BRAM resources by using them for bitstream storage. First, some utilization numbers for comparison: Module LUTs Debug framework (without any module active) (1) 469 BRAM debug tool (module) (2) 344 Logic analyzer (module) (3) 345 LUT and flip-flop altering capability (module)* (4) 390 Configuration readback (module)* (5) 190 Signal statistics (module)* (6) 250 (Wishbone) bus interface (module)* (7) 170 Debug framework with the BRAM debug tool added (8) 674 The LUT utilization figures for module names denoted with a * are estimates, calculated in section (8) is the system presented in chapter 7, that is, basically (1) with (2) integrated, with bus macros etc. removed. Comparing the usage numbers for (1) plus (2) with (8) should give a good indication of how much the overhead the logic required for PR is: = 139. That is, 139 extra LUTs for the ability to perform partial reconfiguration (with 15 bus macros). This is an interesting result, since only the bus macros alone should occupy = 240 LUTs. Either are the logic for the bus macros not considered in the map utilization summary (not very likely), or the situation is such that the signals would have occupied LUTs anyway, and now they happened to use the LUTs in the bus macros instead. With such a small overhead it is clear that it sure is beneficial to use partial reconfiguration as long as at least two modules are going to be used, provided that the bitstreams are stored on a computer. In order to know how large the bitstream will be, the PRR area must be determined. As an example the BRAM debug tool as a module occupies 229 slices (this might not be the optimal solution, it is possible that the design can fit in less slices). Simple calculations give that

64 50 Results 229 slices should fit in an area 14 columns wide and 20 rows in height. However, some slices are occupied by the bus macros, there are two types of slices (SLI- CEL and SLICEM) and PRR areas can only fall on CLB boundaries[21], which may force a larger designated area. The slices between X0Y86 and X11Y109 is the smallest possible area (that is 12 slices wide and begins at X0Y86) which is achievable for this design. This PRR boundary can be seen in figure 8.1. The Figure 8.1: Screenshot of FPGA editor showing a PRR with darkened background. The bus macros can be seen crossing the boundary on the right side. bitstream for this area is bytes in size. If the bitstream is to be stored in BRAMs, 72505/2352 = 31 BRAMs are needed. Also, logic is needed to perform the internal configuration and handling the memories. Note that the estimated logic usage for (4) is much higher than for this example. This clearly shows that it is not feasible to store the bitstreams locally in the FPGA; it is just not acceptable to expect so much resources to be free in the device. After all, this is a design that should use low amounts of resources in order to fit in the device together with the UUT. For the smaller modules it may be possible to use bitstream compression. However, minor tests have shown that not much is gained by the simple compression algorithm used. Unused logic in the designated PRR The area for a PRR is defined in the user constraints file. It must be at least as large as the largest PRM that is going to be implemented in it. There is a risk of wasting logic if not the largest PRM is used in a design, unless the PRR dimensions are adjusted accordingly.