1. OCAU Merchandise is available! Check out our 20th Anniversary Mugs, Classic Logo Shirts and much more! Discussion in this thread.
    Dismiss Notice

AI / neural networks / language models

Discussion in 'General Software' started by ShadowBurger, Apr 7, 2023.

  1. RnR

    RnR Member

    Oct 9, 2002
    Are there standardised tests you can run on each model to get a feel of their pro's and con's?
  2. mtma

    mtma Member

    Aug 12, 2009
    There are some standardised tests, though you'll have to go through the details and rieu detre to figure out if they're relevant to you because most of them are very subject global in nature. If you're using off the shelf LLM you'll have to evaluate it against your context - a balance of capability, implementation and resource consumption. For example if you're using an LLM to assist in code prompt generation as a natural language interface then you'll be interested in things like token limits.

    I'm sure there's a more concise resource though the recently hyped up BitNet b1.58 paper calls out a few different test suites that they use to evaluate their model against. The b1.58 part is also interesting for the point that they're claiming to have made a promising implementation that completely deletes what GPU's were designed to do best from the necessary hardware to operate a capable LLM, though I don't believe any of their trained models are public at the moment for one to try.

    Also bonus video from something I viewed and thought was an interesting demonstration of "prompt engineering" being used. (beware: title is spicier sounding than the actual way it's done)

Share This Page