Nvidia releases weights for Llama-3.1-Nemotron-70B-Instruct

I gave it a softball question and it got it horribly wrong.

"Cidr /17 is what subnet mask?"

Here is it's answer:

"A simple subnetting question!

Given: CIDR /17

To determine the corresponding subnet mask, we need to understand how CIDR notation works. CIDR (Classless Inter-Domain Routing) notation is a compact way to express the relationship between an IP address and its associated routing prefix. The number after the slash (/) represents the number of leading 1-bits in the subnet mask when it's converted to binary.

A subnet mask for IPv4 is 32 bits long. Here's how to convert /17 to a subnet mask:

Determine the number of 1-bits: /17 means the first 17 bits are 1. Determine the number of 0-bits: Since the total length is 32 bits, and 17 are 1-bits, the remaining bits are 0. So, 32 - 17 = 15 bits are 0. Convert to binary and then to dotted decimal: Binary: The first 17 bits are 1, and the last 15 bits are 0. 11111111111111111 000000000000000 Dotted Decimal: Divide the binary into four parts (each 8 bits long, known as octets) and convert each to decimal. 11111111 = 255 11111111 = 255 00000000 = 0 000000000000000 (last 8 bits of the 15 zeros) = 0 So, the subnet mask for CIDR /17 in dotted decimal notation is: 255.255.0.0."

Yes it's a simple subnetting question and it got it wrong. Right answer is 255.255.128.0.

> As of 1 Oct 2024, this model is #1 on all three automatic alignment benchmarks (verified tab for AlpacaEval 2 LC), edging out strong frontier models such as GPT-4o and Claude 3.5 Sonnet.

> this model can correctly the question How many r in strawberry? without specialized prompting or additional reasoning tokens

Can be tested here: https://huggingface.co/chat/models/nvidia/Llama-3.1-Nemotron...