MyHDL Discourse

Pass large intbv to helper function


#1

Hi,

I’m trying to write a block, that does some calculation with large integers (as intbv), which should be implemented in some helper functions. How do I properly write that with MyHDL? Here’s a snippet:

from __future__ import print_function
from myhdl import *

# FIXME: GHDL: Empty interface list not allowed
def Sha256_new():
    # big-endian: first word contains high-order bits
    return modbv(0x6a09e667bb67ae853c6ef372a54ff53a510e527f9b05688c1f83d9ab5be0cd19)[256:]

# FIXME: MyHDL: Variable has same name as a hierarchical Signal
def Sha256_transform(outDig, inDig, inBlock):
    outDig = inDig

# FIXME: MyHDL: Param type is unconstrained unsigned
def Sha256_transform2(inDig, inBlock):
    res = inDig[:]
    res[32*6:32*5] = 0
    return inDig

@block
def Sha256(ioDigest, inBlock, reset):
    @instance
    def logic():
        while True:
            yield ioDigest, inBlock, reset

            if reset != 1:
                ioDigest.next = Sha256_new()
            else:
                # TODO: How to write something like that?
                # Sha256_transform(ioDigest.next, ioDigest, inBlock)
                ioDigest.next = Sha256_transform2(ioDigest, inBlock)

    return logic

inst = Sha256(
    ioDigest = Signal(modbv(0)[256:]),
    inBlock  = Signal(modbv(0)[512:]),
    reset    = Signal(bool(1))
)

inst.convert(hdl='VHDL', path='work')
inst.convert(hdl='Verilog', path='work', no_testbench=True)

I use the VHDL conversion (up-to-date GitHub version of MyHDL):

library IEEE;
use IEEE.std_logic_1164.all;
use IEEE.numeric_std.all;
use std.textio.all;

use work.pck_myhdl_10.all;

entity Sha256 is
    port (
        ioDigest: inout unsigned(255 downto 0);
        inBlock: in unsigned(511 downto 0);
        reset: in std_logic
    );
end entity Sha256;

architecture MyHDL of Sha256 is

function MYHDL4_Sha256_new(
    ) return unsigned is
begin
    return unsigned'("0110101000001001111001100110011110111011011001111010111010000101001111000110111011110011011100101010010101001111111101010011101001010001000011100101001001111111100110110000010101101000100011000001111110000011110110011010101101011011111000001100110100011001");
end function MYHDL4_Sha256_new;

function MYHDL5_Sha256_transform2(
    inDig: in unsigned;
    inBlock: in unsigned
    ) return unsigned is
    variable res: unsigned(255 downto 0);
begin
    res := inDig;
    res((32 * 6)-1 downto (32 * 5)) := to_unsigned(0, 32);
    return inDig;
end function MYHDL5_Sha256_transform2;

begin

SHA256_LOGIC: process (ioDigest, inBlock, reset) is
begin
    if (reset /= '1') then
        ioDigest <= ;
    else
        ioDigest <= MYHDL5_Sha256_transform2(ioDigest, inBlock);
    end if;
end process SHA256_LOGIC;

end architecture MyHDL;

There are some problems:
[1] The call of Sha256_new is missing in the code. The Verilog output seems to look ok.
[2] The parameters and return type of MYHDL5_Sha256_transform2 remain unconstrained. Is that ok? Will such a function be synthesizeable?

Apparently, I’m new to hardware design languages. Also I’m struggling with wiring up the block. How would MyHDL code look like, that uses such a block as a sub-routine, i.e., assigns signals and waits for the result. Right now, my simulation seems to end prematurely.

Thanks in advance,
Mario


#2

May I suggest to start with something simpler then. SHA-256 is reasonably complex and there are architectural trade-offs to be made between size and speed of the resulting implementation. And it is a sizeable amount of work. Do have an effective application-case for this SHA-256? If so can you describe the requirements?

Regards,

Josy


#3

You’re right, SHA-256 isn’t the best algorithm to start with. Unfortunately, this isn’t anything I can change. A cryptographically strong hash algorithm is the requirement. Also, I looked through the examples on the MyHDL website and wrote some simpler ones to get some experience first.

In fact, the full SHA-256 is already working also using MyHDL types/classes. I’m an experienced programmer, just no hardware design yet, which is why I thought MyHDL may be helpful to increase testability/prototyping.

Having that said, my problem is the actual conversion to VHDL, in particular, those helper functions written in Python
to get the algorithm implemented. The posted example was just a stripped-down showcase. For me it’s not that trivial to decide if those are MyHDL bugs or just not supported Python features…

In general, is it ok to use self-written Python functions calling other such functions (no recursion) from within HDL blocks?


#4

Does it need to work in hardware? If so, is there a speed/throughput requirement? Or a maximum size of the device?

[quote]
In fact, the full SHA-256 is already working also using MyHDL types/classes.[/quote]
I take it that you can simulate your code? Can we see that code?

Yes that is OK. Even recursion should work, although MyHDL may not (yet) allow so.
I see a few candidates in SHA-256 where I would use a function too: e.g. the rightrotate (in Wikipedia’s pseudo-code)

I ran your code through my local MyHDL code-base (which is a heavily evolved branch from the pre-@block tree). And you have a case: the VHDL conversion should make distinction between functions with and without arguments. One without should convert to:

function functionName  return returntype is
    begin
        ....
        return returnvalue; 
    end function functionName;

The other issue, the omission of the function instantiation is also an error in to _VHDL.py, the code simply expects there is at least one argument.
Both issues are easy to solve.
But in defence: the use case for a VHDL function without arguments is very rare … In your particular case you could replace the Sha256_new() function with a simple constant.


Add support for nested functions
Add support for nested functions
#5

Thanks for your help and sorry for not replying for a while, but I was busy doing other projects…

Yes. There is of course a size requirement, but that’s not an issue right now. I might either optimize the algorithm to fit or we might use another piece of hardware. It’s still in the process.

Not if you mean a MyHDL simulation. I never tried. But I ran the algorithm and verified its correctness against python’s internal hashlib. If you are interested, I can also post the full source code. However, for brevity, below is just another shortened version.

Exactly. That was also my intention, but there are some problems. First the code:

HW.py:

#!/usr/bin/env python

# vim: set tabstop=4 shiftwidth=4 expandtab:

from __future__ import print_function
from myhdl._Signal import _Signal

import myhdl as HW
import sys


# set long to int for Python 3 to work
if sys.version_info[0] >= 3:
    long = int


# MyHDL: http://docs.myhdl.org/en/latest/manual/index.html
class Component(object):
    '''
        Base class for MyHDL components (entities) defining
        some helper functions for convenience.
    '''
    @staticmethod
    def makeSignal(val, width=None, **kwd):
        '''
            Converts value :val: into a `Signal` assuring a width of :width:
            if specified and it is an `intbv` or `modbv`.
            Uses the value itself if it is already an instance of `Signal`.
        '''
        if not isinstance(val, _Signal):
            # skip bool here to not slice it below
            if isinstance(val, (int, long)) and not isinstance(val, bool):
                val = HW.intbv(val)
            # fix width if not yet sized
            if isinstance(val, (HW.intbv, HW.modbv)) and not len(val):
                assert width, "Need width to slice bit-vector"
                val = val[width:]
            # make signal
            val = HW.Signal(val, **kwd)

        # always check size if specified
        assert (len(val) == width) or (width is None), "Invalid signal width"
        return val


def dump(val):
    print(type(val), ':', sep='')
    if isinstance(val, (HW.intbv, HW.modbv)):
        print(' ', vars(val))


# FIXME: Workaround for VHDL/Verilog output name derived from method name
def _block(funcOrName):
    '''
        Annotates a function as hardware block optionally
        specifying a new name for VHDL/Verilog code generation.
    '''
    if isinstance(funcOrName, str):
        def block(func):
            func.__name__ = funcOrName
            return HW.block(func)
        return block
    else:
        return HW.block(funcOrName)


# finally import remaining symbols
from myhdl import *

# ... and redefine block
block = _block

Sha256.py:

#!/usr/bin/env python

# vim: set tabstop=4 shiftwidth=4 expandtab:

# PYTHON_ARGCOMPLETE_OK

from __future__ import print_function
# FIXME: MyHDL: need direct import, doesn't convert as HW.modbv()
from HW import intbv, modbv

import hashlib
import mando
import sys
import HW

# ********************************************************************************
# ************************ Low-level SHA-2 implementation ************************
# ********************************************************************************

def D(x):
    return modbv(x)[32:]

def BUG(x):
    return x

# FIXME: GHDL: Empty interface list not allowed
def Sha256_new(dummy=False):
    # big-endian: high-order bits contain first word
    return modbv(0x6a09e667bb67ae853c6ef372a54ff53a510e527f9b05688c1f83d9ab5be0cd19)[256:]

# FIXME: MyHDL: Param type is unconstrained unsigned
def Sha256_transform(inDig, inBlock):
    # prepare input data
    W = [D(0) for i in range(64)]

    for i in range(64):
        if i < 16:
            # copy input
            W[i][:] = inBlock[32*(i+1):32*i]
        else:
            # FIXME: MyHDL: assigning 0 works, a function call does not
            W[i][:] = BUG(0)

    # TODO: replace mock with real implementation
    return inDig

def Sha256_getDigestWords(digest):
    # big-endian: high-order bits contain first word
    return tuple(digest[32*(i+1):32*i] for i in range(7, -1, -1))

def Sha256_transform_buffer(digest, input):
    assert len(input) == 64, "Invalid block length"
    charAt  = lambda i: ord(input[i]) if isinstance(input, str) else int(input[i])
    inBlock = intbv()[512:]

    for i in range(16):
        inBlock[32*(i+1):32*i] = (
            (charAt(i*4+0) << 24) +
            (charAt(i*4+1) << 16) +
            (charAt(i*4+2) <<  8) +
            (charAt(i*4+3) <<  0)
        )

    digest[:] = Sha256_transform(digest, inBlock)

# ********************************************************************************
# ******************************* MyHDL component ********************************
# ********************************************************************************

class Sha256(HW.Component):
    def __init__(self, ioDigest=None, inBlock=None, reset=None):
        self.ioDigest = self.makeSignal(ioDigest or 0, 256)
        self.inBlock  = self.makeSignal(inBlock  or 0, 512)
        self.reset    = self.makeSignal(reset    or False)

    @HW.block('Sha256')
    def make(self):
        @HW.instance
        def logic():
            while True:
                yield self.ioDigest, self.inBlock, self.reset

                if self.reset != 1:
                    self.ioDigest.next = Sha256_new(False)
                else:
                    self.ioDigest.next = Sha256_transform(self.ioDigest, self.inBlock)

        return logic

# ********************************************************************************
# ********************************** Test code ***********************************
# ********************************************************************************

Script = mando.core.Program(fromfile_prefix_chars='@')

def testImpl(impl, hash, shaTrans, shaDigest):
    print(impl + ':')

    for i in range(3):
        shaTrans(hash)
        print(i, ' '.join('{:#010x}'.format(int(w)) for w in shaDigest(hash)))
        print(i, '0x' + ''.join('{:08x}'.format(int(w)) for w in shaDigest(hash)))

@Script.command
@Script.arg('impl', choices=('hashlib', 'sha256', 'all'))
def test(impl='all'):
    '''
        Test hash implementations.
        :param -i, --impl ALGO: hash implementation, one of: %(choices)s (default: %(default)s)
    '''
    Message = ''.join(chr(i) for i in range(64))
    pad = ''

    # call hashlib.sha256() multiple times
    if impl.lower() in ('hashlib', 'all'):
        testImpl(
            pad + "hashlib",
            hash      = hashlib.__get_builtin_constructor('sha256')(),
            shaTrans  = lambda hash: hash.update(Message),
            shaDigest = lambda hash: hash._sha['digest']
        )
        pad = '\n'

    # test new Sha256 implementation
    if impl.lower() in ('sha256', 'all'):
        testImpl(
            pad + "Sha256",
            hash      = Sha256_new(),
            shaTrans  = lambda hash: Sha256_transform_buffer(hash, Message),
            shaDigest = lambda hash: Sha256_getDigestWords(hash)
        )
        pad = '\n'

@Script.command
def generate(workdir='.'):
    '''
        Generate VHDL/Verilog model.
        :param -w, --workdir DIR: output directory (default: %(default)s)
    '''
    # FIXME: Cannot call bound method here: Missing ports
    inst = Sha256.make(Sha256())
    inst.convert(hdl='VHDL', path=workdir, architecture='RTL')
    inst.convert(hdl='Verilog', path=workdir, no_testbench=True)

if __name__ == '__main__':
    sys.exit(Script())

HW.py contains some helper code, I usually prefer to use. Sha256.py contains the actual implementation with just the function Sha256_transform() stripped down. There are 2 problems:

[1] The parameter types of Sha256_transform() are unconstrained integers in VHDL, but not in Verilog:

function MYHDL5_Sha256_transform(
    inDig: in unsigned;
    inBlock: in unsigned
    ) return unsigned is
    type t_array_W is array(0 to 64-1) of unsigned(31 downto 0);
    variable W: t_array_W;
begin
    -- ...
end function MYHDL5_Sha256_transform;
function [256-1:0] MYHDL15_Sha256_transform;
    input [256-1:0] inDig;
    input [512-1:0] inBlock;
    integer i;
    reg [32-1:0] W [0:64-1];
begin: MYHDL20_RETURN
    // ...
end
endfunction

[2] The function call to BUG() at line 42 raises an exception:

/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py:360: ToVHDLWarning: Output port is read internally: self_ioDigest
  category=ToVHDLWarning
Traceback (most recent call last):
  File "./NewSha256.py", line 146, in <module>
    sys.exit(Script())
  File "/usr/local/lib/python2.7/site-packages/mando/core.py", line 197, in __call__
    return self.execute(sys.argv[1:])
  File "/usr/local/lib/python2.7/site-packages/mando/core.py", line 193, in execute
    return command(*a)
  File "./NewSha256.py", line 142, in generate
    inst.convert(hdl='VHDL', path=workdir, architecture='RTL')
  File "/usr/local/lib/python2.7/site-packages/myhdl/_block.py", line 276, in convert
    return converter(self)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 244, in __call__
    _convertGens(genlist, siglist, memlist, vfile)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 538, in _convertGens
    v.visit(tree)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1331, in visit_Module
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1642, in visit_FunctionDef
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1255, in visit_If
    self.mapToIf(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1321, in mapToIf
    self.visit_stmt(node.else_)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1524, in visit_stmt
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 973, in visit_Assign
    self.visit(rhs)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1105, in visit_Call
    v.visit(node.tree)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1331, in visit_Module
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1901, in visit_FunctionDef
    self.visit_stmt(node.body)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1524, in visit_stmt
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1235, in visit_For
    self.visit_stmt(node.body)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1524, in visit_stmt
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1255, in visit_If
    self.mapToIf(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1321, in mapToIf
    self.visit_stmt(node.else_)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1524, in visit_stmt
    self.visit(stmt)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 973, in visit_Assign
    self.visit(rhs)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/site-packages/myhdl/conversion/_toVHDL.py", line 1104, in visit_Call
    v = Visitor(node.tree, self.funcBuf)
AttributeError: '_ConvertFunctionVisitor' object has no attribute 'funcBuf'

And I don’t really understand why. Of course assigning a fixed value would also work, but that’s just a mock to show the problem. That’s somehow a real show-stopper right now… FYI: This is MyHDL git trunk.

You’re right. It’s indeed a corner case. Meanwhile, I solved the issue by adding a dummy parameter. However, if you’re new to MyHDL but not programming in general, it’s a bit unexpected and I felt it’s worth to report it. Also, please note that assigning a fixed value would have worked, but it’s not reusable and giving that value a name (i.e., a global constant) does not work. That’s the reason I decided to define a simple function instead.