Jackie, I don't know of any material that has actually been sold for under rugs and carpeting that is like this, but I can make a guess. Sounds like it might be the safety tread, swimming pool tread, boat deck tread material that someone used for this. If it is, this material came on rolls in various sizes and stuck down like any tape. How well it sticks depends on how well the floor was prepped, but you can probably get a corner loose with a putty knife (a plastic putty knife would stand less chance of scratching the floor) and then you can pull the tape off. You will probably have some adhesive left on the floor and Goof Off should get that up.
If this doesn't sound right, let us know and we can guess again. Seeing it would help but your description sounds pretty much like this material.